Open timblaktu opened 4 years ago
nervo
commented
4 years ago Hello @timblaktu :)
As this role handle aptly package installation and configuration, it needs root access, and the become flag, if needed in your situation, is welcomed.
To run commands as the aptly user, we use su, i guess that's where your issue come from.
Could you try to log in as root, and type su aptly -c "aptly repo list --raw" ?
timblaktu
commented
4 years ago Same error, when running your suggested command as root:
root@aptly:/home/ansible# su aptly -c "aptly repo list --raw"
ERROR: mkdir $HOME: permission deniedI don't know what is trying to mkdir $HOME here. But note that $HOME var for root user is /root:
root@aptly:/home/ansible# echo $HOME
/rootwhich already exists and is owned by root:root, with 700 mode.
I'm running the playbook which invokes your role as a normal user with sudoer privileges, using become: true, and the default become method is sudo.
nervo
commented
4 years ago i suppose the permission denied is about aptly user home, not root one. Does it have a home ? What is its permissions ?
timblaktu
commented
4 years ago That's not the problem, aptly home exists and is writeable by aptly and root (of course):
ansible@aptly:~$ ls -la /home
total 16
drwxr-xr-x 4 root root 4096 Oct 30 20:10 .
drwxr-xr-x 18 root root 4096 Oct 30 11:23 ..
drwxr-xr-x 7 ansible ansible 4096 Nov 3 09:06 ansible
drwxr-xr-x 3 aptly users 4096 Oct 30 20:35 aptlyI found something interesting in aptly home, looks like something created a directory named '$HOME':
ansible@aptly:~$ ls -la /home/aptly
total 32
drwxr-xr-x 3 aptly users 4096 Oct 30 20:35 .
drwxr-xr-x 4 root root 4096 Oct 30 20:10 ..
drwxr-xr-x 3 aptly users 4096 Oct 30 20:35 '$HOME'
-rw-r--r-- 1 root root 499 Oct 30 20:32 .aptly.conf
-rw------- 1 aptly users 160 Oct 30 20:36 .bash_history
-rw-r--r-- 1 aptly users 220 Apr 17 2019 .bash_logout
-rw-r--r-- 1 aptly users 3526 Apr 17 2019 .bashrc
-rw-r--r-- 1 aptly users 807 Apr 17 2019 .profileOk, that '$HOME' is weird... it looks like an escaping issue...
Oh, and what about this .aptly.conf owned by root ?
timblaktu
commented
4 years ago As you can see in the playbook snippet I posted above, I specify:
manala_aptly_user: aptly
manala_aptly_config_file: "/home/{{ manala_aptly_user }}/.aptly.conf" and although I have become: true commented out, I originally ran this with it uncommented at first. I don't know if that explains why that .aptly.conf is owned by root.
Let me know if you have any ideas. I will now reset the aptly VM to the pre-provisioning snapshot I have, and then rerun the ansible playbook against it with become: true and see how far it now gets.
nervo
commented
4 years ago Ah, yes, i did'nt realize you have changed the default config file path, that's why .aptly.conf is owned by root, you should not do that :) Fyi, here is an exemple of a fully functionnal aptly provisionning:
aptly_user: aptly
aptly_config:
- rootDir: /mnt/data/debian
- architectures:
- amd64
aptly_repositories:
- name: stretch
comment: Stretch
component: main
distribution: stretch
origin: Foo
label: Bar
- name: buster
comment: Buster
component: main
distribution: buster
origin: Foo
label: BarAnd that's all :)
timblaktu
commented
4 years ago Yeah, I saw the example in your README. Are you saying that your documented role variable manala_aptly_config_file is not supposed to be used? Please confirm.
If so, I can experiment with using the default instead, but I strongly recommend you either:
BTW, I reran my playbook on fresh Debian buster machine (with become: true) and reproduced the same problem. I will now rerun again using default aptly config location.
timblaktu
commented
4 years ago Using you role's default for aptly config location, I still get the same error, so something else is going on:
TASK [manala.aptly : repositories > Create] **************************************************************************************************failed: [aptly] (item={u'comment': u'\\"Packages for Debian Stretch\\"', u'distribution': u'stretch', u'component': u'main', u'name': u'b-stretch'}) => changed=true
ansible_loop_var: item
cmd:
- su
- aptly - -c
- aptly repo create -comment="Packages for Debian Stretch" -component=main -distribution=stretch b-stretch delta: '0:00:00.013134'
end: '2020-11-03 13:28:22.245441'
item: comment: \"Packages for Debian Stretch\"
component: main
distribution: stretch name: b-stretch
msg: non-zero return code
rc: 1 start: '2020-11-03 13:28:22.232307'
stderr: 'ERROR: mkdir $HOME: permission denied'
stderr_lines: <omitted>
stdout: '' stdout_lines: <omitted>There is no longer a $HOME file in aptly home, and here is the other pertinent info:
aptly@aptly:~$ ls -la
total 28
drwxr-xr-x 4 aptly users 4096 Nov 3 13:31 .
drwxr-xr-x 4 root root 4096 Nov 3 13:25 ..
-rw-r--r-- 1 aptly users 220 Apr 17 2019 .bash_logout
-rw-r--r-- 1 aptly users 3526 Apr 17 2019 .bashrc
drwx------ 3 aptly users 4096 Nov 3 13:31 .gnupg
-rw-r--r-- 1 aptly users 807 Apr 17 2019 .profile
drwx------ 2 aptly users 4096 Nov 3 13:31 .ssh
aptly@aptly:~$ ls -la /etc/aptly.conf
-rw-r--r-- 1 root root 499 Nov 3 13:28 /etc/aptly.conf
aptly@aptly:~$ ls -la /home
total 16
drwxr-xr-x 4 root root 4096 Nov 3 13:25 .
drwxr-xr-x 18 root root 4096 Oct 30 11:23 ..
drwxr-xr-x 5 ansible ansible 4096 Nov 3 13:31 ansible
drwxr-xr-x 4 aptly users 4096 Nov 3 13:31 aptly(Note that the aptly default location for this file is ~/.aptly.conf, per https://www.aptly.info/doc/configuration.)
While I have your attention (thank you, BTW!) is there any support in your role for setting up https access and http forwarding using ssl cert files and an nginx reverse proxy?
nervo
commented
4 years ago Well, what can i say... :) This variable acts as a default value, if you know that you have to change it for any reasons, feel free to do it. If you're unsure, treat it like a default value, and don't change it.
nervo
commented
4 years ago Btw, this role is atomic, and have the responsability to install and configure, and only install and configure aptly. The rest is your own responsibility :)
nervo
commented
4 years ago The fact is there are some tests, and i can't see exactly what can differs from your own playbook...
Could you run them ? make test.debian.buster for instance
timblaktu
commented
4 years ago Below are the test results. It looks like apt is encountering some fetch errors. (i didn't see related errors when running my playbook, btw.)
tblack-stretch]cm/ansible/projects/aptly > find . -name Makefile
./roles/manala.aptly/.manala/make/Makefile
./roles/manala.aptly/Makefile
[tblack-stretch]cm/ansible/projects/aptly > cd roles/manala.aptly
[tblack-stretch]projects/aptly/roles/manala.aptly > make test.debian.buster
[22:26:38] [test@local] Test "manala.aptly" on "debian.buster"
[22:26:38] [test@local] tests/0100_install.yml
Unable to find image 'manala/test-ansible:debian.buster' locally
debian.buster: Pulling from manala/test-ansible
31dd5ebca5ef: Pull complete
593a7cf29bc2: Pull complete
c54ca6e117a6: Pull complete
fdf7f86f3a36: Pull complete
bb74ef7c30e8: Pull complete
1f48375c11f3: Pull complete
3a5b2b0e40f3: Pull complete
2d2f9d7cab45: Pull complete
536c38f53bcf: Pull complete
1104aa07dcb7: Pull complete
97ef5d46315e: Pull complete
Digest: sha256:90dd040746bdb20d1806779261a107217d7e5ff30cb4cae2d390b2894ed97d6b
Status: Downloaded newer image for manala/test-ansible:debian.buster
/usr/local/lib/python2.7/dist-packages/cryptography/__init__.py:39: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
CryptographyDeprecationWarning,
PLAY [0100_install] **************************************************************************************************************************
TASK [Gathering Facts] ***********************************************************************************************************************ok: [debian.buster]
TASK [Pre tasks > Aptly apt key] *************************************************************************************************************changed: [debian.buster]
TASK [Pre tasks > Aptly apt repository] ******************************************************************************************************An exception occurred during task execution. To see the full traceback, use -vvv. The error was: apt.cache.FetchFailedException
fatal: [debian.buster]: FAILED! => {
"changed": false,
"rc": 1
}
MSG:
MODULE FAILURE
MODULE_STDERR:
Traceback (most recent call last):
File "/tmp/ansible_ueXBbK/ansible_module_apt_repository.py", line 550, in <module>
main()
File "/tmp/ansible_ueXBbK/ansible_module_apt_repository.py", line 542, in main
cache.update()
File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 591, in update
raise FetchFailedException()
apt.cache.FetchFailedException
PLAY RECAP ***********************************************************************************************************************************debian.buster : ok=2 changed=1 unreachable=0 failed=1
make: *** [.manala/make/Makefile:176: tests/0100_install.yml] Error 2
[22:28:23] [test@local] tests/0200_config.yml
/usr/local/lib/python2.7/dist-packages/cryptography/__init__.py:39: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
CryptographyDeprecationWarning,
PLAY [0200_config] ***************************************************************************************************************************
TASK [Gathering Facts] ***********************************************************************************************************************ok: [debian.buster]
TASK [Pre tasks > Aptly apt key] *************************************************************************************************************changed: [debian.buster]
TASK [Pre tasks > Aptly apt repository] ******************************************************************************************************An exception occurred during task execution. To see the full traceback, use -vvv. The error was: apt.cache.FetchFailedException
fatal: [debian.buster]: FAILED! => {
"changed": false,
"rc": 1
}
MSG:
MODULE FAILURE
MODULE_STDERR:
Traceback (most recent call last):
File "/tmp/ansible_D9c0Bn/ansible_module_apt_repository.py", line 550, in <module>
main()
File "/tmp/ansible_D9c0Bn/ansible_module_apt_repository.py", line 542, in main
cache.update()
File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 591, in update
raise FetchFailedException()
apt.cache.FetchFailedException
PLAY RECAP ***********************************************************************************************************************************debian.buster : ok=2 changed=1 unreachable=0 failed=1
make: *** [.manala/make/Makefile:176: tests/0200_config.yml] Error 2
[22:29:35] [test@local] tests/0300_repositories.yml
/usr/local/lib/python2.7/dist-packages/cryptography/__init__.py:39: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
CryptographyDeprecationWarning,
PLAY [0300_repositories] *********************************************************************************************************************
TASK [Gathering Facts] ***********************************************************************************************************************ok: [debian.buster]
TASK [apt] ***********************************************************************************************************************************fatal: [debian.buster]: FAILED! => {
"changed": false
}
MSG:
No package matching 'gnupg1' is available
PLAY RECAP ***********************************************************************************************************************************debian.buster : ok=1 changed=0 unreachable=0 failed=1
make: *** [.manala/make/Makefile:176: tests/0300_repositories.yml] Error 2
.manala/make/Makefile:193: recipe for target 'test@local' failed
make: *** [test@local] Error 1Wtf...
Ok, forget about the tests.
The issue does not come from the role itself, but from the command su aptly -c "aptly repo list --raw". I guess aptly tries to write in its home folder, but get $HOME environement variable from root.
Could you try su aptly -c "env" ?
timblaktu
commented
4 years ago env and aptly help commands work via su aptly -c, but no aptly repo commands work (see below).
Remember, I'm running the playbook which invokes your role as a normal user named ansible that has passwordless sudoer privilege, and now I'm using become: true at the play level, and using the default become method sudo. When I run the playbook I pass it --extra-vars "ansible_user=ansible ansible_ssh_pass=*** ansible_become_pass=***".
How is this different from how you run the playbook successfully? What become_method is your role assuming?
ansible@aptly:~$ su aptly -c "env"
Password:
SHELL=/bin/bash
PWD=/home/ansible
LOGNAME=aptly
XDG_SESSION_TYPE=tty
_=/usr/bin/env
HOME=/home/aptly
LANG=en_US.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
SSH_CONNECTION=172.16.15.107 37400 172.16.22.24 22
XDG_SESSION_CLASS=user
TERM=screen-256color
USER=aptly
SHLVL=1
XDG_SESSION_ID=45
XDG_RUNTIME_DIR=/run/user/1000
SSH_CLIENT=172.16.15.107 37400 22
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
MAIL=/var/mail/aptly
SSH_TTY=/dev/pts/0
ansible@aptly:~$ su aptly -c "aptly help"
Password:
aptly is a tool to create partial and full mirrors of remote
repositories, manage local repositories, filter them, merge,
upgrade individual packages, take snapshots and publish them
back as Debian repositories.
aptly's goal is to establish repeatability and controlled changes
in a package-centric environment. aptly allows one to fix a set of packages
in a repository, so that package installation and upgrade becomes
deterministic. At the same time aptly allows one to perform controlled,
fine-grained changes in repository contents to transition your
package environment to new version.
Options:
-architectures="": list of architectures to consider during (comma-separated), default to all available
-config="": location of configuration file (default locations are /etc/aptly.conf, ~/.aptly.conf)
-db-open-attempts=10: number of attempts to open DB if it's locked by other instance
-dep-follow-all-variants: when processing dependencies, follow a & b if dependency is 'a|b'
-dep-follow-recommends: when processing dependencies, follow Recommends
-dep-follow-source: when processing dependencies, follow from binary to Source packages
-dep-follow-suggests: when processing dependencies, follow Suggests
-dep-verbose-resolve: when processing dependencies, print detailed logs
-gpg-provider="": PGP implementation ("gpg" for external gpg or "internal" for Go internal implementation)
ansible@aptly:~$ su aptly -c "aptly repo list --raw"
Password:
ERROR: mkdir $HOME: permission deniedThe role (is supposed to) don't care about become method or become user.
As we can see, although $HOME is ok, the aptly repo list --raw seems to do something badly... But what ?
I think you should open an issue on aptly repository :)
timblaktu
commented
4 years ago Thanks. I filed an issue with aptly repo.
When creating that issue, I noticed that my aptly installation, done by this role, shows "unknown version":
ansible@aptly:~$ aptly version
aptly version: unknown
ansible@aptly:~$ su aptly -c "aptly version"
Password:
aptly version: unknown
ansible@aptly:~$ which aptly
/usr/bin/aptlyI'm using all the defaults in your role. I see in your install task, that this is just installing aptly package. On my Debian Buster system, this version apt shows is: 1.3.0+ds1-2.2~deb10u1.
Looks like 1.4.0 is the latest tag on the aptly git project. The 1.4.0 changelog doesn't seem to address this issue though.
nervo
commented
4 years ago Well, you're installing aptly package from debian repositories, and not aptly ones. Let's see how aptly maintainers address your issue :)
timblaktu
commented
4 years ago OK thanks. I also wanted to check with you if I am supposed to be creating the aptly user before invoking your role. Because, I am, using this play run before your role:
- name: Configure aptly user and groups
tags: ["user"]
become: true
hosts: aptly_server
tasks:
- name: Create groups for aptly user
group:
name: "{{ item.name }}"
with_items:
- { name: aptly }
- name: >
Create aptly user if it doesn't exist, with default home dir and specified groups.
This is a Linux system user that aptly processes will run as.
user:
name: "{{ aptly_linux_user_name }}"
password: $6$T4ZTzdc.wu/sm$zoYfpWS41hrv0dXWFy/J9eN/cEAsnSMC3kPfmMgCLu17XEJXjt.TpMFYd9ucpZ8tXtbf3Ai9v/.x06FjOwMN0
groups:
- aptly
- wheel
shell: /bin/bashRegarding the dir named "$HOME" was created in /home/aptly:
ansible@aptly:~$ ls -la ~aptly
total 36
drwxr-xr-x 5 aptly users 4096 Nov 4 13:43 .
drwxr-xr-x 4 root root 4096 Nov 3 13:25 ..
drwxr-xr-x 3 aptly users 4096 Nov 4 13:43 '$HOME'
-rw------- 1 aptly users 717 Nov 9 09:59 .bash_history
-rw-r--r-- 1 aptly users 220 Apr 17 2019 .bash_logout
-rw-r--r-- 1 aptly users 3526 Apr 17 2019 .bashrc
drwx------ 3 aptly users 4096 Nov 9 10:00 .gnupg
-rw-r--r-- 1 aptly users 807 Apr 17 2019 .profile
drwx------ 2 aptly users 4096 Nov 3 13:31 .sshI noticed that it appears to contain the expected contents in aptly user's home dir, i.e.
ansible@aptly:~$ tree -a ~aptly/\$HOME/
/home/aptly/$HOME/
└── .aptly
├── db
│ ├── 000006.ldb
│ ├── 000009.log
│ ├── CURRENT
│ ├── LOCK
│ ├── LOG
│ └── MANIFEST-000010
└── public
├── dists
│ └── stretch
│ ├── main
│ │ └── binary-amd64
│ │ ├── Packages
│ │ ├── Packages.bz2
│ │ ├── Packages.gz
│ │ └── Release
│ └── Release
└── pool
8 directories, 11 files
When running the aptly role I am getting "ERROR: mkdir $HOME: permission denied" in the
aptly repo listandaptly repo createcalls made in tasks/repositories.yml.I am able to ssh into the controlled aptly server as the aptly user and run the same commands, without su, successfully. I believe the problem is the su commands used to execute aptly commands.
I am using
manala_aptly_user: aptlyand I get the same error with and without running the role withbecome: true.There is no instruction in the role docs for what ansible user to run as, and there are runtime warnings about the su call:
How am I supposed to get past this user issue? Here is how my playbook invokes your role:
I'm using role version 2.0.2 and ansible version 2.9.10:
ansible 2.9.10 config file = /home/tblack/src/cm/ansible/projects/aptly/ansible.cfg configured module search path = ['/home/tblack/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /home/tblack/.local/lib/python3.8/site-packages/ansible executable location = /home/tblack/.local/bin/ansible python version = 3.8.3 (default, Jun 30 2020, 11:18:52) [GCC 6.3.0 20170516]
Thanks for your time!