Unable to write to file using script in container

[hawkspar]

Expected Behavior

I'm expecting spython.main.Client.execute to behave exactly like singularity exec from the command line when running a dummy python script to write to a file.

Actual Behavior

I can create a file using the command line but not using spython.main.Client.execute from inside a python script.

Steps to Reproduce

Consider running singularity exec --bind ./local_dir:instance_dir any_image_with_python.sif python -c "with open('instance_dir/test.tx','w') as f: f.write('test')" and compare the result with the same command in a python script :

from spython.main import Client

Client.execute('any_instance_with_python.sif',["python","-c","with open('instance_dir/test.tx','w') as f: f.write('test')"],bind=[./local_dir:instance_dir`])

I am confident these are supposed to be identical because I've run the above with the quiet optional argument set to False. Yet one created a file, the other doesn't.

Context

• singularity version: 4.1.4-1
• spython version: 0.3.13
• python version: 3.9.18

Failure Logs

I would love to have those. The python script just continues without a sound...

Possible Fix

My guess is this is a permission issue which requires increasing permissions for the script ?

[vsoch]

It's probably something to do with how the command is executed - perhaps try running that via a script instead?

[hawkspar]

You mean a bash script ? That's a good idea. I'll also try running the python command from the python shell.

On Thu, 15 Aug 2024, 02:14 Vanessasaurus, @.***> wrote:

It's probably something to do with how the command is executed - perhaps try running that via a script instead?

— Reply to this email directly, view it on GitHub https://github.com/singularityhub/singularity-cli/issues/222#issuecomment-2290112178, or unsubscribe https://github.com/notifications/unsubscribe-auth/APA6REP3UZHGFQHIYAIMPO3ZRPXGFAVCNFSM6AAAAABMQFWARGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJQGEYTEMJXHA . You are receiving this because you authored the thread.Message ID: @.***>

[hawkspar]

Hello all,

As stated before, I originally ran a python script.

The problem remains when running a test case on the command line with the -c option.

I wrote a one liner bash script looking like python3 -c "with open(...". This script produces a file when running singularity exec on it and doesn't, without raising an error, if ran using Client.execute from the spython library.

I have no clue where this could be coming from. The commands are supposed to be identical...

On Thu, 15 Aug 2024, 09:04 Quentin Chevalier, @.***> wrote:

You mean a bash script ? That's a good idea. I'll also try running the python command from the python shell.

On Thu, 15 Aug 2024, 02:14 Vanessasaurus, @.***> wrote:

It's probably something to do with how the command is executed - perhaps try running that via a script instead?

— Reply to this email directly, view it on GitHub https://github.com/singularityhub/singularity-cli/issues/222#issuecomment-2290112178, or unsubscribe https://github.com/notifications/unsubscribe-auth/APA6REP3UZHGFQHIYAIMPO3ZRPXGFAVCNFSM6AAAAABMQFWARGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJQGEYTEMJXHA . You are receiving this because you authored the thread.Message ID: @.***>

[vsoch]

As suggested, you need to write into a script. The library here uses subprocess, which expects a list of commands (using shlex split) and here is what is happening:

# From the command line
 python -c "print('hello')"

hello

From Python

import subprocess
p = subprocess.Popen(['python', '-c', "'print(\"hello\")'"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.communicate()
# note that there is no output
(b'', b'')

The function is here: https://github.com/singularityhub/singularity-cli/blob/2b222339382e92f9b93357b3bc81d6558e4efdcd/spython/utils/terminal.py#L158-L167. If you'd like to suggest a way to use subprocess to get your desired output, I'd be happy to review a PR or put in a quick one myself. I suspect there is some kind of fork or other so it's not picked up here, but I haven't looked into it. The only way I can grep to see output (not capture it) would be to do:

os.system('python -c \"print(\'hello\')\"')
hello

But that is doing os.system and would not be good to put into the library. Thanks.