"Not a directory" when trying to install role containing .class file

[bcbrockway]

We have a role stored in git which contains a compiled java .class file in the files/ directory but when I try and run an ansible-galaxy install -r requirements.yml -p /path/to/roles/dir --force i get an exception:

- extracting onetick_base to /root/ans_onetick/roles/onetick_base
ERROR! Unexpected Exception, this is probably a bug: [Errno 20] Not a directory: u'/root/ans_onetick/roles/onetick_base/files/udf/CMC_VWAP.class'
the full traceback was:

Traceback (most recent call last):
  File "/root/ansible/bin/ansible-galaxy", line 109, in <module>
    exit_code = cli.run()
  File "/root/ansible/lib/ansible/cli/galaxy.py", line 150, in run
    self.execute()
  File "/root/ansible/lib/ansible/cli/__init__.py", line 155, in execute
    fn()
  File "/root/ansible/lib/ansible/cli/galaxy.py", line 393, in execute_install
    installed = role.install()
  File "/root/ansible/lib/ansible/galaxy/role.py", line 322, in install
    role_tar_file.extract(member, self.path)
  File "/usr/lib64/python2.7/tarfile.py", line 2084, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name))
  File "/usr/lib64/python2.7/tarfile.py", line 2160, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "/usr/lib64/python2.7/tarfile.py", line 2200, in makefile
    with bltn_open(targetpath, "wb") as target:
IOError: [Errno 20] Not a directory: u'/root/ans_onetick/roles/onetick_base/files/udf/CMC_VWAP.class'

The directory structure of the role looks like this:

.
|-- defaults
|   `-- main.yml
|-- files
|   `-- udf
|       |-- CMC_VWAP.class
|       `-- CMC_VWAP$CMC_VWAP_proc.class
|-- meta
|   `-- main.yml
|-- README.md
`-- tasks
    `-- main.yml

I've done some testing with Docker and pyenv. With Python 2.6 there's no exception, but the .class files are not there and "udf" is a file instead of a directory. With Python 2.7 I get the above exception and it stops processing requirements.yml. A normal git clone pulls the repo down as expected.

Running Centos 7.4.1708 in Docker container (same as awx 1.0.2). Here's the Dockerfile I'm using:

FROM centos:centos7.4.1708

WORKDIR /root

# ENV http_proxy="PROXY_HERE_IF_REQUIRED"
# ENV https_proxy="PROXY_HERE_IF_REQUIRED"

# Box setup:
RUN yum install -y \
    bzip2 \
    bzip2-devel \
    gcc \
    git \
    libffi-devel \
    make \
    openssl-devel \
    patch \
    python-devel \
    readline-devel \
    sqlite \
    sqlite-devel \
    which \
    xz \
    xz-devel \
    zlib-devel
RUN yum install -y epel-release
RUN yum install -y python-pip
RUN yum remove -y epel-release
RUN pip install --upgrade pip

# Pyenv setup
ENV PYENV_ROOT="/root/.pyenv"
ENV PATH="/root/.pyenv/bin:$PATH"
RUN git clone https://github.com/pyenv/pyenv.git ~/.pyenv
RUN echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
RUN echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
RUN echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.bashrc

# Ansible setup
RUN git clone https://github.com/ansible/ansible.git
RUN pip install -r /root/ansible/requirements.txt

# Copy SSH over in case of SSH git repo:
#
# RUN mkdir /root/.ssh
# RUN chmod 700 ~/.ssh
# COPY ./id_rsa /root/.ssh/id_rsa
# RUN chmod 600 /root/.ssh/id_rsa

Then just...

docker build . -t ansible_issue
docker run -it ansible_issue /bin/bash
pyenv install <VERSION>
export PYENV_VERSION=<VERSION>
pip install -r /root/ansible/requirements.txt
source /root/ansible/hacking/env-setup

[atsalolikhin-spokeo]

I hit this issue as well. I'm using ansible-galaxy [core 2.14.5].

[atsaloli]

I found this is due to the dollar sign character in the file name.

I was able to duplicate this issue with ansible-galaxy [core 2.15.2] using the following requirements.yml:

---
- src: https://gitlab.com/atsaloli/ansible-galaxy-issue-271.git
  version: main
  scm: git

Console log:

$ ansible-galaxy install -r requirements.yml --force
Starting galaxy role install process
- extracting ansible-galaxy-issue-271 to /home/atsaloli/.ansible/roles/ansible-galaxy-issue-271
[WARNING]: - ansible-galaxy-issue-271 was NOT installed successfully: Could not update files in /home/atsaloli/.ansible/roles/ansible-galaxy-issue-271:
[Errno 20] Not a directory: '/home/atsaloli/.ansible/roles/ansible-galaxy-issue-271/files/udf/CMC_VWAP.class'
ERROR! - you can use --ignore-errors to skip failed roles and finish processing the list.
$

I made a change to my installation of Ansible, I edited role.py to remove the check for dollar sign in the name part of the file:

$ diff /home/atsaloli/ansible-galaxy-venv/lib/python3.11/site-packages/ansible/galaxy/role.py.bak /home/atsaloli/ansible-galaxy-venv/lib/python3.11/site-packages/ansible/galaxy/role.py
379c379
<                                     if n_part != '..' and not n_part.startswith('~') and '$' not in n_part:
---
>                                     if n_part != '..' and not n_part.startswith('~'):
$

After this edit, I can install the reproducible example role.

[atsaloli]

Here is the line in question: https://github.com/ansible/ansible/blob/01469c558c2533604d7eb93b91883438bd2621a6/lib/ansible/galaxy/role.py#L379

[atsaloli]

I'm a bit puzzled as commit 1c83672532330d0d26b63f8a97ee703e7fc697df has the summary:

Allow `$` & `~` inside paths in galaxy roles (#72966)

but it only fixes ~.

Here is the original story: https://github.com/ansible/ansible/issues/72966

The pull request https://github.com/ansible/ansible/pull/73372 has the title "Allow ~ inside paths in galaxy roles #73372"

[atsaloli]

Earlier work on this part of the code:

• https://github.com/ansible/ansible/issues/69133
• https://github.com/ansible/ansible/pull/69213

[atsaloli]

I am having trouble finding where the $ check came from originally.

[:~/git/ansible/ansible/lib/ansible/galaxy] [ansible-galaxy-venv] devel 130 ± git log -S 'not in n_part'
commit 1c83672532330d0d26b63f8a97ee703e7fc697df
Author: Alexander Sowitzki <asowitzk@redhat.com>
Date:   Tue Feb 2 18:10:05 2021 +0100

    Allow `$` & `~` inside paths in galaxy roles (#72966)

    ansible-galaxy currently behaves bad then a role to be installed
    contains ~ or $ at any place in the path of a file in that role.
    It extracts the parent directory of the offending path level as an
    empty file. This explodes if that directory contains anything else.

    Change this behaviour. `~` is now allowed allowed when it is
    not a full level (Yes: `some~thing/`, no: `~/`). The code should
    get refactoring in an other PR.

commit bc41dd45141f627b31fbb04443fbaa1ea0c7d1d4
Author: Matt Martz <matt@sivel.net>
Date:   Tue Apr 28 15:33:44 2020 -0500

    Handle non-ascii paths during role installation. Fixes #69133 (#69213)
[:~/git/ansible/ansible/lib/ansible/galaxy] [ansible-galaxy-venv] devel ±

Aha, looks like it was added in commit ce3ef7f4c1 not in part

I don't see it in the Web UI but I see it on the git CLI.

This doesn't tell me why we have a check for $.

Okay, I found the commit that added the check: https://github.com/ansible/ansible/commit/f8845af1951fb3745acddb0696fd988810719a0b

I still don't understand why we are checking for $.

[atsalolikhin-spokeo]

@jimi-c Can you shed any light on why you added a check for $ in https://github.com/ansible/ansible/commit/f8845af1951fb3745acddb0696fd988810719a0b, please?

[atsalolikhin-spokeo]

@eqrx Can you comment on why $ fell out of scope of https://github.com/ansible/ansible/pull/73372 please?

[eqrx]

Hey @atsalolikhin-spokeo, thanks for ping. I honestly do not know anymore though, sorry.

[bcoca]

This is not the right repo for ansilbe-galaxy CLI issues, those should go to https://github.com/ansible/ansible

The Core team members (like jimi-c) have had to setup ignores for most repos that they are not actively maintaining, like this one (galaxy team handles things here), that is why you had no answers from him.

While the error itself is correct, that is not a file name Ansible will accept, it should not be an unhandled exception. So I still consider this a bug, but the resolution won't be to accept files with $ in the name as not only can it create other errors, it also poses a security issue, many times paths read are also evaluated and it would look like an environment variable.

[atsalolikhin-spokeo]

@bcoca I've opened https://github.com/ansible/ansible/issues/81553 to move this issue to the right repo. Could you please close this issue?