Permission error using singularity image

[haraldgrove]

I tried running PAV comparing a pig assembly to the reference, using the example command (I was running it using slurm):

singularity run --bind "$(pwd):$(pwd)" --writable-tmpfs library://becklab/pav/pav:latest -c 16

and got the following error message:

INFO:    Downloading library image
…
Traceback (most recent call last):

  File "/usr/local/lib/python3.11/site-packages/snakemake/__init__.py", line 575, in snakemake
    workflow = Workflow(
               ^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/snakemake/workflow.py", line 242, in __init__
    self.sourcecache = SourceCache()
                       ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/snakemake/sourcecache.py", line 334, in __init__
    os.makedirs(self.cache, exist_ok=True) 
  File "<frozen os>", line 215, in makedirs 
  File "<frozen os>", line 215, in makedirs 
  File "<frozen os>", line 215, in makedirs 
  [Previous line repeated 1 more time] 
  File "<frozen os>", line 225, in makedirs 
PermissionError: [Errno 13] Permission denied: '/home/default'

Is this a problem with the singularity file or with our computing environment?

-Harald

[paudano]

The --writable-tmpfs Singularity option should be creating a writable overlay in the container, including /home/default. Something PAV uses writes a cache file to the home directory, and it crashed for me before I used that command-line option.

Looks like some systems don't support overlays:

https://docs.sylabs.io/guides/3.6/admin-guide/installation.html#overlay-support

https://github.com/apptainer/singularity/issues/5461

I'll see if I can figure out a better way to work-around whatever is caching to the home directory. I didn't realize this would be a limitation on some systems.

[paudano]

I probably will not be able to work on this today, but I'll set aside time next week for this issue.

[joanocha]

Hi! I have also encountered this error. So I decided to try another route while waiting for it to be fixed and do the native install. I am currently running pav with runlocal and with the example you provided. It works well for the most part, but there are a lot of errors for rules that use pandas (e.g. using append instead of pd.concat with dataframes, among others). I started by hard coding the rules to make it work but perhaps it would be best if you could specify which pandas version we should use? Or update the code so it runs on latest python and pandas versions? Thanks

[paudano]

I was able to fix the image so it doesn't need the writable tempfs. There's a little work to do, and I'm testing now. I should be able to release a new version by the end of tomorrow.

The singularity image is using Pandas 1.5.2. There are some deprecated features that need to be worked out, and those will generate warnings (on my radar to fix). What version are you using?

[paudano]

Version 2.2.7 has been released. Run without --writable-tmpfs, it's no longer needed.

If you have any difficulty running this, please re-open this issue.

[joanocha]

I still get the permission error without --writable-tmpfs. And now with the new version I get this error with runlocal: No module named 'svpoplib' File "/global/scratch/users/joana_rocha/software/pav/Snakefile", line 66, in File "/global/scratch/users/joana_rocha/software/pav/pavlib/init.py", line 3, in File "/global/scratch/users/joana_rocha/software/pav/pavlib/asmstat.py", line 10, in

[paudano]

On the local install, is there anything in dep/svpop (relative to the PAV install directory, dep is in the same location as Snakefile)?

In Snakefile, these lines add dep/svpop and its dependencies to the library search path:

sys.path.append(PIPELINE_DIR)  # pavlib
sys.path.append(os.path.join(PIPELINE_DIR, 'dep', 'svpop'))  # svpoplib
sys.path.append(os.path.join(PIPELINE_DIR, 'dep', 'svpop', 'dep'))  # kanapy
sys.path.append(os.path.join(PIPELINE_DIR, 'dep', 'svpop', 'dep', 'ply'))  # ply - lexer / parser

...Where PIPELINE_DIR is the location of the PAV install directory.

As for the permission error, you are definitely using pav:2.2.7, right? If you are using "latest", try explicitly asking for "2.2.7" just to make sure. Can you send me the whole error message again, maybe something else is writing to the home directory that I don't know about.

Can you try making a directory in the run directory (e.g. singularity_tmpfs) and running with that as an overlay (add --overlay singularity_tmpfs/ to the singularity command)? That should be a workaround until I can figure out what's trying to write to $HOME or implement a better solution for it.

[haraldgrove]

I've had some time to try the options and using the 2.2.7 version unfortunately gave the samme error message as before.

Running it with the --overlay option caused some other issues: FATAL: container creation failed: while setting overlay session layout: only root user can use sandbox as overlay in setuid mode While there seems to be some options to get around this, I'm not sure what is currently possible in our system.

[paudano]

If you just run 2.2.7 without any special options, it's still not running? Things writing in the home directory are redirected to /dev/shm, and it shouldn't require overlay or anything to run.

If something else is writing to the home directory and it's still failing, I'll need to set something up so I can monitor it better. It's going to be mid-July before I can do that though, I have a big deadline I have to meet before then and probably won't be able to set aside time for PAV before then. My apologies for the delays!

[haraldgrove]

I've managed to get it to run and, I'm fairly certain, solved the issue I had. It seems to be an issue with how (our?) Singularity is configured.

I just added the option -H $PWD:/home to the singularity command.

This was based on the answer to a question on Stackoverflow about auto binding with singularity.

I don't know if this is the best solution, but I'll have to leave it for someone else to decide that.

[paudano]

I am glad it's working, and this is helpful feedback. I'd still like to figure out what's writing to home and find a better way around that. I'll leave this open for now, but I don't need anything unless you want to try testing it later.

Thank you!

[paudano]

I improved the code that was writing to home, and I believe this is fixed now. All the updates will be out in 2.3.1. Thank you!

[joanocha]

Thank you! Sorry I totally lost this thread! I forgot to say I managed to work with it a while ago via singularity instead of runlocal and it has been working fine!

On Tue, Sep 19, 2023 at 4:53 PM Peter Audano @.***> wrote:

I improved the code that was writing to home, and I believe this is fixed now. All the updates will be out in 2.3.1. Thank you!

— Reply to this email directly, view it on GitHub https://github.com/EichlerLab/pav/issues/35#issuecomment-1725936462, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKH557YS4GP4ONUGQZEI4YTX3G5XNANCNFSM6AAAAAAYHVBF6E . You are receiving this because you commented.Message ID: @.***>