Pyff tries to write downloaded metadata to /var/run/pyff

[baszoetekouw]

I'm running a number of pyff pipelines in batch modes as unprivileged user. This results in warnings like:

Nov 07 10:55:06 docker pyff-metadata[16709]: ERROR:pyff.utils:[Errno 30] Read-only file system: '/var/run/pyff'

in the logs.

It is unclear what pyff is doing in /var/run. Only after about an hour of debugging, I found that it is trying to write a copy of the downloaded metadata to /var/run/pyff/backup (from here: https://github.com/IdentityPython/pyFF/blob/master/src/pyff/resource.py#L465). It is unclear why it is doing that, and why it insist on doing that in /var/run (doesn't seem to be configurable).

Code Version

2.1.1

Expected Behavior

Pyff should not save a backup by default, should not write to /var/run unless it is running as pyffd, and the location where this backup is saved should be configurable. Also, it would be extremely helpful for debugging if Pyff would tell me which file it is trying to write, if an error occurs during the write.

Current Behavior

Pyff writes a copy of downloaded metadata (i.e., from a remote url that is specified in a load pipeline) to /var/run/pyff/backup

Possible Solution

Don't write the backup file at all when running in batch mode.

Steps to Reproduce

Use this pipeline:

- load:
    - https://metadata.surfconext.nl/signed/2023/idps-metadata.xml
- select
- finalize:
    cacheDuration: P7D
    validUntil: P14D
- publish: "/tmp/test.xml"
- stats

Running it as s non-privileged user gives:

ansible@docker:/tmp$ ./pyff-env/bin/pyff test.fd
ERROR:pyff.utils:[Errno 13] Permission denied: '/var/run/pyff'
---
total size:     207
selected:       207
          idps: 207
           sps: 0
---

[c00kiemon5ter]

Check this part from the configuration https://github.com/IdentityPython/pyFF/blob/8c03d504277483fafb0a85a486fd5386979ab379/src/pyff/constants.py#L230-L234

You can set PYFF_LOCAL_COPY_DIR env var to point to a different location.

[leifj]

Yep. This is expected behaviour. Set the cache directory parameter to somwhere writable.

[baszoetekouw]

ahh, I hadn't found that. Still, I think the default of using /var/run for this is not very intuitive; something like /var/cache would make more sense.

Anyway, from the error message alone, it is quite hard to understand what exactly is going wrong. I would suggest at least improving that.

[leifj]

Point taken.

[baszoetekouw]

And is there also a way to disable caching altogether? The help text suggests

 --caching_enabled             enable caching? [True]

But neither of these is accepted:

ansible@docker:/opt/metadata$ ./pyff-env/bin/pyff --caching_enabled false test.fd
ansible@docker:/opt/metadata$ ./pyff-env/bin/pyff --caching_enabled False test.fd
ansible@docker:/opt/metadata$ ./pyff-env/bin/pyff --caching_enabled 0 test.fd
ansible@docker:/opt/metadata$ ./pyff-env/bin/pyff --caching_enabled=false test.fd
ansible@docker:/opt/metadata$ ./pyff-env/bin/pyff --caching_enabled=False test.fd
ansible@docker:/opt/metadata$ ./pyff-env/bin/pyff --caching_enabled=0 test.fd
ansible@docker:/opt/metadata$ ./pyff-env/bin/pyff --no-caching_enabled test.fd
ansible@docker:/opt/metadata$ ./pyff-env/bin/pyff --caching_disabled test.fd

[leifj]

-C