database.sh Push/Pull/Backup failing

davebowker commented 6 years ago

Submit a feature request or bug report

[x ] I've read the guidelines for Contributing to this project in the README.md
[ ] This is a feature request
[ x] This is a bug report
[x ] This request isn't a duplicate of an existing issue
[x ] I've read the README and followed its instructions (if applicable)

Replace any X with your information.

Hey :-),

Trying to do a backup from prod to dev and all commands are failing, include push pull and backup on the database.sh command.

I ran this: ./bin/database.sh production [sitename] pull and I also tried this: ./bin/database.sh production [sitename] backup

Got this:

System info: Ansible 2.5.4; Darwin Trellis version (per changelog): "build-before: Checkout project source code to local temporary directory"

The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'wordpress_sites'

The error appears to have been in '/Users/debe/Sites/roots/trellis/database- pull.yml': line 48, column 7, but may be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- name: Pull database dump from {{ env }} to development
  ^ here

We could be wrong, but this one looks like it might be an issue with missing quotes. Always quote template expression brackets when they start a value. For instance:

with_items:
  - {{ foo }}

Should be written as:

with_items:
  - "{{ foo }}"

fatal: [REDACTED]: FAILED! => {} to retry, use: --limit @/Users/debe/Sites/roots/trellis/database-pull.retry

PLAY RECAP ***** 178.128.175.19 : ok=4 changed=1 unreachable=0 failed=1

Any help greatly appreciated. Not sure if I missed something or if it's a problem caused by new updates? Thanks

D

UPDATE So, I have ran the uploads.sh script and it connects and pulls from the prod server successfully. This leads me to believe that the setup of the hosts alias is correct.

Though any command on the database.sh script fails here: TASK [Pull database dump from production to development]

Also for push and backup

Thanks

zsimoes commented 6 years ago

I also have this problem, with the same symptoms. I'm on Ansible 2.4.0; Darwin.

I can also push and pull uploads, but not the DB, ansible fails with the error described above.

-- UPDATE --

Upgraded to ansible 2.6.1, the problem is still there.

valentinocossar commented 6 years ago

Hi, I can't reproduce the issue. I've tested the tool with the latest Ansible versions (up to version 2.6.1) and it works. From what I see, the hosts configuration seems to be correct, maybe the issue could be related with the project folders structure or the wordpress_sites.yml configuration. Could you share the project structure and your wordpress_sites.yml configuration (for all the environments)? Obviously, remember to hide or replace sensible data.

zsimoes commented 6 years ago

@valentinocossar Sure. I have no staging for this project. Here's Dev and Prod:

Development - wordpress_sites.yml

wordpress_sites:
  <hidden>.pt:
    site_hosts:
      - canonical: <hidden>.local
        redirects:
          - www.<hidden>.local
    local_path: ../site
    admin_email: <hidden>
    multisite:
      enabled: false
    ssl:
      enabled: false
      provider: self-signed
    cache:
      enabled: false

Production - wordpress_sites.yml

wordpress_sites:
  <hidden>.pt:
    site_hosts:
      - canonical: <hidden>.pt
        redirects:
          - www.<hidden>.pt
    local_path: ../site
    repo: git@gitlab.com:<hidden>.git
    repo_subtree_path: site
    branch: master
    multisite:
      enabled: false
    ssl:
      enabled: false
      provider: letsencrypt
    cache:
      enabled: false

And my host configuration, just in case:

Development

# Trellis database sync:
development_host ansible_host=192.168.50.5 ansible_connection=ssh ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/default/virtualbox/private_key ansible_ssh_extra_args="-o IdentitiesOnly=yes -o StrictHostKeyChecking=no  -o GlobalKnownHostsFile=/dev/null -o UserKnownHostsFile=/dev/null"

[development]
192.168.50.5 ansible_connection=local

[web]
192.168.50.5 ansible_connection=local

Production

production_host ansible_host=<hidden ip>

[production]
<hidden ip>

[web]
<hidden ip>

--- EDIT ---

Ansible 2.6.1 Darwin, Latest Trellis master branch version Commit no 411981f

My error log:

TASK [Create database dump on development] *****************************************************************************************************************************************************************
changed: [<hidden ip> -> 192.168.50.5]

TASK [Push database dump from development to production] ***************************************************************************************************************************************************
System info:
  Ansible 2.6.1; Darwin
  Trellis version (per changelog): "Update xdebug tunnel configuration"
---------------------------------------------------
The task includes an option with an undefined variable. The error was: 'dict
object' has no attribute 'wordpress_sites'

The error appears to have been in
'/Users/.../trellis/database-push.yml': line 49,
column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

    - name: Push database dump from development to {{ env }}
      ^ here
We could be wrong, but this one looks like it might be an issue with
missing quotes.  Always quote template expression brackets when they
start a value. For instance:

    with_items:
      - {{ foo }}

Should be written as:

    with_items:
      - "{{ foo }}"

valentinocossar commented 6 years ago

Trellis version? macOS version? Are you running ./bin/database.sh production <hidden>.pt push inside Trellis root folder? Thank you!

zsimoes commented 6 years ago

Latest Trellis master branch. MacOS High Sierra 10.13.6, and yes, I'm running that command in the trellis folder.

valentinocossar commented 6 years ago

I don't see anything strange and if you can connect to the servers with ssh web@<hidden production ip> and ssh vagrant@192.168.50.5 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -i .vagrant/machines/default/virtualbox/private_key -o ForwardAgent=yes, I have to ask you to try to provide me further debug data from Ansible, because I can't replicate the issue.

This is the only thing that is different from what I get running ./bin/database.sh production domain.com push:

TASK [Create database dump on development] *****************************************************************************************************************************************************************
changed: [<hidden ip> -> 192.168.50.5]

For me is this:

TASK [Create database dump on development] *******************************************************************************************************************************************************************
changed: [production_host -> 192.168.50.13]

zsimoes commented 6 years ago

Strange. I'll try to dig deeper in a few days, when I have more time. Maybe you can close this for now if you'd like, until (if) I can find something. Cheers, thanks!

valentinocossar commented 6 years ago

I leave this issue open, let me know if you find other details.

handpressed commented 6 years ago

I'm having the exact same issue as @zsimoes - running Ubuntu 18.04 LTS locally. No additional details at this time - will dig deeper.

handpressed commented 6 years ago

Fixed it, @zsimoes and @valentinocossar. Add the host to the relevant group and not to the top of the file, e.g.:

Development:

[development]
development_host ansible_host=192.168.50.5 ansible_connection=ssh ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/default/virtualbox/private_key ansible_ssh_extra_args="-o IdentitiesOnly=yes -o StrictHostKeyChecking=no  -o GlobalKnownHostsFile=/dev/null -o UserKnownHostsFile=/dev/null"
192.168.50.5 ansible_connection=local

[web]
192.168.50.5 ansible_connection=local

Production:

[production]
production_host ansible_host=<hidden ip>
<hidden ip>

[web]
<hidden ip>

Edited out second issue - I switched my development site to self-signed SSL.

handpressed commented 6 years ago

@valentinocossar - The handful of changes I made to get everything to work on my Trellis setup are in my fork. I can submit a pull request if you're happy with them and they work for others.

zsimoes commented 6 years ago

@handpressed fantastic, thanks.

valentinocossar commented 6 years ago

@handpressed Awesome. I've seen your fork and found that the host is declared only for the first group. Are you sure this is the right choice? If yes, could you prove it in some way? Just to be sure we do things in the right way (according to Ansible's point of view).

Open a new pull request so we can discuss it in a proper space (and I can test it). Thank you!

handpressed commented 6 years ago

@valentinocossar - No problem. Whether it's the right choice or not, I'm not sure. It works for me and @zsimoes. Interestingly, I tried your original hosts config setup and I didn't receive any errors this time, which is frustrating.

The only difference between the two is the cli output - the original outputs e.g.:

TASK [Delete database dump from development] *******************************************************
changed: [production_host -> 192.168.50.5]

... whereas my alternative config outputs:

TASK [Delete database dump from development] *******************************************************
changed: [1.2.3.4 -> 192.168.50.5]

... where 1.2.3.4 is your_server_hostname. Both configs seem to work fine for me.

I've added my config to the README.md file of my fork as an alternative - if users receive an error message like the one reported in this thread, they can try it.

I'll submit a pull request shortly.

davebowker commented 6 years ago

@handpressed I've just tried your solution, with the updated command that's currently in the readme file, and it all works well now, for both database and uploads!

Thanks for your help! :-)

handpressed commented 6 years ago

@davebowker - You're welcome. I've used both database.sh and uploads.sh extensively with the hosts config listed in my fork's README for two days without issue. Syntactically there was nothing wrong with @valentinocossar's original config, but if my changes work for those having issues then maybe it's the best config to use.

valentinocossar commented 6 years ago

Hi @davebowker, I've just released version 1.0.1 with an improved configuration for the hosts and other small fixes and typos. I and @handpressed have tested it before the release, please let me know if the new configuration works well also for you. Thank you!

valentinocossar / trellis-database-uploads-migration