search

Donders-Institute / bidscoin

BIDScoin converts your source-level neuroimaging data to BIDS
https://bidscoin.readthedocs.io
GNU General Public License v3.0
130 stars 35 forks source link

Can path to .bidscoin be configured? #226

Closed mateuszpawlik closed 9 months ago

mateuszpawlik commented 9 months ago

General summary I've just realized that the newer BIDScoin hard-codes the path to .bidscoin directory to reside in home. What if I don't want it there?

Additional detail I came across that when executing BIDScoin (dicomsort tool) in a container without home mounted.

Next steps It's not up to me. Maybe at least an option to configure that? If not, make it more visible in the documentation?

marcelzwiers commented 9 months ago

It's stated when you run:

$ bidscoin
usage: bidscoin [-h] [-l] [-p] [-i INSTALL [INSTALL ...]] [-u UNINSTALL [UNINSTALL ...]] [-d DOWNLOAD] [-t [TEST]] [-b BIDSMAPTEST]
                [--tracking {yes,no,show}] [-v]

BIDScoin is a toolkit to convert and organize raw data-sets according to the Brain Imaging Data Structure (BIDS)

The basic workflow is to run these two tools:

  $ bidsmapper sourcefolder bidsfolder        # This produces a study bidsmap and launches a GUI
  $ bidscoiner sourcefolder bidsfolder        # This converts your data to BIDS according to the study bidsmap

Default settings and template bidsmaps are stored in the ".bidscoin" folder in your home directory (you can modify
them to your needs with any plain text editor)

It's also stated in the docs (at least in the latest), but that can always be improved of course. The path to the config is now defined like this (in __init__.py):

# Create a BIDScoin user configuration directory if needed and load the BIDScoin user settings
configfile     = Path.home()/'.bidscoin'/__version__/'config.toml'
templatefolder = configfile.parent/'templates'

I don't see an easy way to configure the path to the config :-p. Where would you like it? Note that the user specific templates are copied from the central install, so if you change it there then all your users will have that version (and they can modify it individually). Or what is your use case?

marcelzwiers commented 9 months ago

I guess I can add a few lines to check if home exists, otherwise use the central location

mateuszpawlik commented 9 months ago

My use case is a fully containerized conversion pipeline. In Nextflow, the home is not mounted by default and that causes the problem. I can mount it, so you don't have to change anything. I simply wasn't aware of these changes and need to document that myself. Wasn't BIDScoin writing to code directory before and is it still doing so?

marcelzwiers commented 9 months ago

It certainly writes to the code directory, all the log are stored there (not ideal, but I want to keep that close to the data and then there is no alternative). In fact, you may perhaps like the new bidscoiner.tsv file in there :-)

marcelzwiers commented 9 months ago

Also note that bidscoiner can now run in parallel. I don't know if you already did so in nextflow, but I guess that would lead to concurrency issues (e.g. multiple workers writing to the same participants.tsv file)

mateuszpawlik commented 9 months ago

I don't see ant tsv file. Which version introduce it? I'm using 4.2.1. I'm executing it for one participant in isolation.

marcelzwiers commented 9 months ago

No, it will be in the upcoming release. Since I plan to release it today or tomorrow, you have to be quick if you want to put something in :-)

mateuszpawlik commented 9 months ago

Oh. Thanks a lot for this kind offer :-)

With config files, I usually see some priority lists, like config in your home, an env variable set to point to your config, config in /etc/..., etc. python-gitlab has a nice example: https://python-gitlab.readthedocs.io/en/stable/cli-usage.html#configuration-files

But I don't want to push this one. I simply didn't want to mount the entire home in the container.

marcelzwiers commented 9 months ago

That's a good example, I can add a check for environment variable that contains the path to the configuration folder (but the folder needs to be writable)

marcelzwiers commented 9 months ago

That was easy :-), you can now use export BIDSCOIN_CONFIG=/writable/path/to/config.toml if you like to use a different config file

mateuszpawlik commented 9 months ago

I've seen other files in .bidscoin directory. Is the change for the config file enough?

/home/mpawlik/.bidscoin/
├── 4.2.1
│   ├── config.toml
│   └── templates
│       ├── bidsmap_dccn.yaml
│       ├── bidsmap_sst.yaml
│       └── schema.json
└── 4.3.0
    ├── config.toml
    ├── templates
    │   ├── bidsmap_bids2bids.yaml
    │   ├── bidsmap_dccn.yaml
    │   ├── bidsmap_sst.yaml
    │   └── schema.json
    └── usage
        └── bidscoin_4a39a33d0ba0040bb243175c70854fa5.db
marcelzwiers commented 9 months ago

I think so, but I haven't tested it :-)

mateuszpawlik commented 9 months ago

It seems the configuration consists of more files.

(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ echo $BIDSCOIN_CONFIG
/home/mpawlik/Remote/anc/anc-data-acquisition/assets/bidscoin_config.toml
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ bidscoin -t assets/bidscoin_map.yaml 
Creating BIDScoin configuration:
-> /home/mpawlik/.bidscoin/4.3.0/config.toml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_sst.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_dccn.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidscoin_map.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_bids2bids.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/schema.json
...
mateuszpawlik commented 9 months ago

I don't like it but I could bind any host directory to $HOME in the container. This should do the trick. Playing with env variables and container is not optimal but I could also pass that.

marcelzwiers commented 9 months ago

It seems the configuration consists of more files.

Yes, but they are all automatically stored in the same folder as the config file. I just tested it and fixed a minor bug (so it should now work)

marcelzwiers commented 9 months ago 
$ export BIDSCOIN_CONFIG=~/tmp/config.toml
$ bidsmapper raw bids
$ tree ~/tmp
/home/marzwi/tmp
├── config.toml
├── templates
│   ├── bidsmap_bids2bids.yaml
│   ├── bidsmap_dccn.yaml
│   ├── bidsmap_sst.yaml
│   └── schema.json
└── usage
    ├── bidscoin_2feb5e843ccb54a41416f963a6c88a5d.bak
    ├── bidscoin_2feb5e843ccb54a41416f963a6c88a5d.dat
    └── bidscoin_2feb5e843ccb54a41416f963a6c88a5d.dir
mateuszpawlik commented 9 months ago

I installed the newest version with pip install --upgrade git+https://github.com/Donders-Institute/bidscoin.git@4f130416305c74d9f7884034e853b89fd6f32ad3 and unfortunately can't reproduce:

(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ echo $BIDSCOIN_CONFIG
/home/mpawlik/Remote/anc/anc-data-acquisition/assets/bidscoin_config.toml
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ rm -r ~/.bidscoin/
rm: cannot remove '/home/mpawlik/.bidscoin/': No such file or directory
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ bidscoin --version
Creating BIDScoin configuration:
-> /home/mpawlik/.bidscoin/4.3.0/config.toml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_sst.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_dccn.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidscoin_map.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_bids2bids.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/schema.json
BIDS-version:           1.9.0
BIDScoin-version:       4.3.0, NB: Your BIDScoin version is NOT up-to-date: 4.3.0 -> 4.2.1
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ tree assets/
assets/
├── a003_bidscoin_map.yaml
├── bidscoin_config.toml
└── bidscoin_map.yaml
marcelzwiers commented 9 months ago

Despite the flag, that doesn't upgrade when installing from github (if the version nr doesn't change), try:

pip install --force-reinstall git+https://github.com/Donders-Institute/bidscoin
mateuszpawlik commented 9 months ago
Well, this errors out now ``` Building wheels for collected packages: bidscoin Building wheel for bidscoin (PEP 517) ... error ERROR: Command errored out with exit status 1: command: /home/mpawlik/Remote/anc/anc-data-acquisition/venv/bin/python3 /tmp/tmpdrqpb91d_in_process.py build_wheel /tmp/tmp30mkamn8 cwd: /tmp/pip-req-build-5evrj7jp Complete output (77 lines): No `packages` or `py_modules` configuration, performing automatic discovery. `flat-layout` detected -- analysing . discovered packages -- ['bidscoin', 'bidscoin.heuristics', 'bidscoin.utilities', 'bidscoin.cli', 'bidscoin.bidsapps', 'bidscoin.plugins', 'bidscoin.schema', 'bidscoin.schema.objects', 'bidscoin.schema.meta', 'bidscoin.schema.rules', 'bidscoin.schema.rules.tabular_data', 'bidscoin.schema.rules.checks', 'bidscoin.schema.rules.files', 'bidscoin.schema.rules.sidecars', 'bidscoin.schema.rules.tabular_data.derivatives', 'bidscoin.schema.rules.files.deriv', 'bidscoin.schema.rules.files.common', 'bidscoin.schema.rules.files.raw', 'bidscoin.schema.rules.sidecars.derivatives'] running bdist_wheel running build running build_py running build_manpages generating man/bidscoin.1 Traceback (most recent call last): File "/tmp/tmpdrqpb91d_in_process.py", line 280, in main() File "/tmp/tmpdrqpb91d_in_process.py", line 263, in main json_out['return_val'] = hook(**hook_input['kwargs']) File "/tmp/tmpdrqpb91d_in_process.py", line 204, in build_wheel return _build_backend().build_wheel(wheel_directory, config_settings, File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 404, in build_wheel return self._build_with_temp_dir( File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 389, in _build_with_temp_dir self.run_setup() File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 311, in run_setup exec(code, locals()) File "", line 3, in File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 103, in setup return distutils.core.setup(**attrs) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup return run_commands(dist) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands dist.run_commands() File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands self.run_command(cmd) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 963, in run_command super().run_command(command) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 368, in run self.run_command("build") File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command self.distribution.run_command(command) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 963, in run_command super().run_command(command) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run self.run_command(cmd_name) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command self.distribution.run_command(command) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 963, in run_command super().run_command(command) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/build_manpages/build_manpages.py", line 151, in run self.run_command(DEFAULT_CMD_NAME) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command self.distribution.run_command(command) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 963, in run_command super().run_command(command) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/build_manpages/build_manpages.py", line 127, in run parser = get_parser(data['import_type'], data['import_from'], data['objname'], data['objtype'], data.get('prog', None)) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/argparse_manpage/tooling.py", line 65, in get_parser return get_parser_from_file(import_from, objname, objtype, prog=prog) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/argparse_manpage/tooling.py", line 52, in get_parser_from_file module_loaded = load_file_as_module(filename) File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/argparse_manpage/compat.py", line 48, in load_file_as_module return load_py_file(filename) File "/usr/lib/python3.9/runpy.py", line 268, in run_path return _run_module_code(code, init_globals, run_name, File "/usr/lib/python3.9/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/usr/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "bidscoin/cli/_bcoin.py", line 28, in from bidscoin import check_version, __version__, bidsversion, bidsmap_template File "./bidscoin/__init__.py", line 79, in bidsmap_template = Path(config['bidscoin']['bidsmap_template']) KeyError: 'bidscoin' ---------------------------------------- ERROR: Failed building wheel for bidscoin Failed to build bidscoin ERROR: Could not build wheels for bidscoin which use PEP 517 and cannot be installed directly ```

uninstall and install also didn't help.

marcelzwiers commented 9 months ago

generating man/bidscoin.1

Ah, that's due to a bug (not mine) when building manpages (see here) Remove that and it should work?

mateuszpawlik commented 9 months ago

A fresh venv didn't help and gave the same error. I did:

(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ deactivate 
mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ rm -rf venv/
mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ python3 -m venv venv                                             
mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ source venv/bin/activate
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ pip install git+https://github.com/Donders-Institute/bidscoin

I can't figure out where are the man pages. This is what I should delete, right?

Any of these?

$ man -w
/usr/local/man:/usr/local/share/man:/usr/share/man
marcelzwiers commented 9 months ago

Yeah, that's not my expertise either, I think it also depends on your distro. In my case they where build in the cwd (CentOS) or somewhere in my venv (OpenSuse). What if you do:

$ find -name bidscoin.1
mateuszpawlik commented 9 months ago

I don't find it anywhere in my home :-( Neither /usr/local/ nor /usr/share/

mateuszpawlik commented 9 months ago

I tried previous commits and pip install git+https://github.com/Donders-Institute/bidscoin.git@8ed9a53ce2202d2ef7520a84eb6642d4e519e91c succeeds without deleting any man pages, which I cannot find anywhere.

mateuszpawlik commented 9 months ago

I don't think it's the ma pages issue here. Here's why installing the newest commits is failing:

684e29367c4e1eeb786a14e1ecc8557397283457

File "./bidscoin/__init__.py", line 66, in <module>                                                                                                                    
      templatefolder = configfile.parent/'templates'                                                                                                                       
  AttributeError: 'str' object has no attribute 'parent'

89fb79644327e7876f100a0ef97d276d477d59c1

File "./bidscoin/__init__.py", line 66, in <module>                                                                                                                    
      templatefolder = configfile.parent/'templates'                                                                                                                       
  AttributeError: 'str' object has no attribute 'parent'

4f130416305c74d9f7884034e853b89fd6f32ad3 (newest)

File "./bidscoin/__init__.py", line 79, in <module>                                                                                                                    
      bidsmap_template = Path(config['bidscoin']['bidsmap_template'])                                                                                                      
  KeyError: 'bidscoin'
mateuszpawlik commented 9 months ago

And this is because my config file pointed by BIDSCOIN_CONFIG env variable was empty and therefore KeyError: 'bidscoin'. It is weird that this error is triggered when installing bidscoin. After filling in my config file, it installs.

marcelzwiers commented 9 months ago

AttributeError: 'str' object has no attribute 'parent'

Yes, I saw and fixed that

https://github.com/Donders-Institute/bidscoin/commit/4f130416305c74d9f7884034e853b89fd6f32ad3 (newest)

That bidscoin key is written in the config.toml file, could it be that writing the config.toml file failed?

marcelzwiers commented 9 months ago

Ah, you set an empty path, yes, that probably fails. Try unset BIDSCOIN_CONFIG

mateuszpawlik commented 9 months ago

No, my config.toml file was empty and therefore the key could not be found in it. Once I added content to my config.toml file, bidscoin installed. I don't know what happens during installation but the bidscoin code is executed.

marcelzwiers commented 9 months ago

I have no idea what happened, but I now added some extra robustness to the code. I hope that solves it :-/

mateuszpawlik commented 9 months ago

No, my config.toml file was empty and therefore the key could not be found in it. Once I added content to my config.toml file, bidscoin installed. I don't know what happens during installation but the bidscoin code is executed.

I'm getting more errors now, like:

$ bidscoin --version
/home/mpawlik/Remote/anc/anc-data-acquisition/venv/lib/python3.9/site-packages/bidscoin/__init__.py:81: RuntimeWarning: Missing template bidsmap: /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_dccn.yaml (see /home/mpawlik/Remote/anc/anc-data-acquisition/assets/bidscoin_config.toml)
  warnings.warn(f"Missing template bidsmap: {bidsmap_template} (see {configfile})", RuntimeWarning)

It still expects files in my home. But the installation copied the template files to the parent directory of BIDSCOIN_CONFIG. It seems that you copy these files during the installation but later expect the path in home.

I'm sorry for causing all this trouble and thank you for taking care of that so quickly.

marcelzwiers commented 9 months ago

No, thanks for testing this, I really appreciate it. I'll have a look

mateuszpawlik commented 9 months ago

I'm wandering why do you need to copy all these files. The config file should be enough to configure bidscoin. If more files are necessary, how about add their paths to the config file instead of copying them.

marcelzwiers commented 9 months ago

In the center where I work, there is a single bidscoin installation that all users share. Copying the default templates to every user's home allows them to edit the default templates to their needs

marcelzwiers commented 9 months ago

It seems that you copy these files during the installation but later expect the path in home.

Only if BIDSCOIN_CONFIG is unset / empty?

mateuszpawlik commented 9 months ago

When BIDSCOIN_CONFIG was set before the installation, the template files were copied to the parent directory of the file pointed by BIDSCOIN_CONFIG. However, errors occurred when executing bidscoin https://github.com/Donders-Institute/bidscoin/issues/226#issuecomment-1941957820

When BIDSCOIN_CONFIG was NOT set before the installation, the installation created ~/.bidscoin directory with all the files.

mateuszpawlik commented 9 months ago

So in my container use case (Apptainer), bidscoin installation in the image created /root/.bidscoin/4.2.1/) directory with all the files. However, when executed (even with --no-home), bidscoin expects the files in my home. Shouldn't bidscoin internally use the path where it copies the files to, instead of $HOME?

Another weird thing is that I get these errors with dicomsort, which does not rely on .bidscoin directory.

I'm also confused now. For bidsmapper, should I use the config file to set my custom map file, or the --template option? What's the priority here?

marcelzwiers commented 9 months ago

Just to make sure your expectations are in line, using the environment variable is not permanent, bidscoin only uses that config file when the variable is non-empty, otherwise it will use the default

marcelzwiers commented 9 months ago

Another weird thing is that I get these errors with dicomsort, which does not rely on .bidscoin directory.

Perhaps that's a bug, let me check

marcelzwiers commented 9 months ago

bidscoin installation in the image created /root/.bidscoin/4.2.1/) directory with all the files.

These file are only created when bidscoin is imported (it is all done in __init__.py), i.e. when a bidscoin function is run. The pip installation itself really should not create anything? Running apptainer should be no problem if you pass the BIDSCOIN_CONFIG every time you run a bidscoin function

marcelzwiers commented 9 months ago

Do you get the same issue if you don't use apptainer?

mateuszpawlik commented 9 months ago

bidscoin installation in the image created /root/.bidscoin/4.2.1/) directory with all the files.

These file are only created when bidscoin is imported (it is all done in __init__.py), i.e. when a bidscoin function is run. The pip installation itself really should not create anything? Running apptainer should be no problem if you pass the BIDSCOIN_CONFIG every time you run a bidscoin function

But it does :thinking:

Bidscoin in Apptainer certainly doesn't remember where it saved the files. It keeps recreating .bidscoin directory if home is not mounted in. But it tries to do that in $HOME and errors out. I solved it with --scratch $HOME Apptainer option.

Relevant Apptainer issue: https://github.com/apptainer/apptainer/issues/388

mateuszpawlik commented 9 months ago

Do you get the same issue if you don't use apptainer?

Is https://github.com/Donders-Institute/bidscoin/issues/226#issuecomment-1941995188 an answer?

marcelzwiers commented 9 months ago

But it does 🤔

Ah, I think building the manpages imports bidscoin, that's when it saves the config file

marcelzwiers commented 9 months ago

Nevertheless, it shouldn't matter, you can have multiple locations where config files are stored. As long as there is nothing in the environment, then the home directory is used, otherwise the location from the environment variable is used. Perhaps there is a weird interaction somewhere that I don't see, but otherwise I don't see how this doesn't work. I'll test a bit more.

mateuszpawlik commented 9 months ago

It seems that my problems were triggered mainly by Apptainer and the hard-coded path to store .bidscoin directory.

I'll try to summarize a bit.

Outside of the Apptainer

It seems to work as intended now. Template files are copied to the parent of BIDSCOIN_CONFIG.

Inside of the container

version 4.2.1

I fix it with --scratch $HOME when Apptainer is executed with --no-home, because bidscoin keeps looking up $HOME although it isn't mounted.

current master version (cd409e1)

Adding --env BIDSCOIN_CONFIG=/root/.bidscoin/4.3.0/config.toml to Apptainer command should help, because bidscoin doesn't know where is .bidscoin without BIDSCOIN_CONFIG set. If it is not set, bidscoin expects .bidscoin in HOME, which for installation in Apptainer is not true (HOME always points to host home).

If config.toml needs to be overwritten, it can be mounted additionally (--bind /path/to/my/config.toml:/root/.bidscoin/4.3.0/config.toml). If BIDSCOIN_CONFIG is set to something else than /root/.bidscoin/4.3.0/config.toml, the parent directory has to be mounted.

In general

I should not need to set BIDSCOIN_CONFIG and care about the .bidscoin directory, especially if I use bidsmapper with --template option. The config file only sets this template file and the tracking. Usually, I can set the options in a config file, or provide them with CLI, with some priorities. Bidscoin could remember where it created .bidscoin directory, whether BIDSCOIN_CONFIG is set or not. Then, it could preset the options from the config file in there, and optionally overwrite them with values provided with CLI. This would make the behavior consistent.

marcelzwiers commented 9 months ago

Inside of the container

version 4.2.1

I fix it with --scratch $HOME when Apptainer is executed with --no-home, because bidscoin keeps looking up $HOME although it isn't mounted.

Ok, that's not good, I will make a fallback-fix for this (in $TMPDIR?)

current master version (cd409e1)

Adding --env BIDSCOIN_CONFIG=/root/.bidscoin/4.3.0/config.toml to Apptainer command should help, because bidscoin doesn't know where is .bidscoin without BIDSCOIN_CONFIG set.

So you pass a config file, that should work, but pointing it to /root puzzles me.

If it is not set, bidscoin expects .bidscoin in HOME, which for installation in Apptainer is not true (HOME always points to host home).

If it's not set, then bidscoin will create it in home, but I'll make that fallback-fix that should cover this

If config.toml needs to be overwritten, it can be mounted additionally (--bind /path/to/my/config.toml:/root/.bidscoin/4.3.0/config.toml). If BIDSCOIN_CONFIG is set to something else than /root/.bidscoin/4.3.0/config.toml, the parent directory has to be mounted.

Sorry, I'm not that used to working with containers, but I don't see why you have to bind that root path? Isn't passing --env BIDSCOIN_CONFIG=/some/mounted/folder/bidscoin/config.toml sufficient?

In general

I should not need to set BIDSCOIN_CONFIG and care about the .bidscoin directory, especially if I use bidsmapper with --template option. The config file only sets this template file and the tracking.

I agree that the config folder shouldn't be a user concern and I'm surely open for suggestions to improve things here..

Usually, I can set the options in a config file, or provide them with CLI, with some priorities.

Well, that sounds great but on the other hand, it is also very common for applications to simply keep their settings in a hidden file or directory in home (e.g. dcm2niix).

Bidscoin could remember where it created .bidscoin directory, whether BIDSCOIN_CONFIG is set or not.

I don't see how bidscoin can remember that without storing it somewhere on disk. Which is what the config file is

Then, it could preset the options from the config file in there, and optionally overwrite them with values provided with CLI. This would make the behavior consistent.

Currently I don't provide a tool or CLI options for editing/overruling the config file, the user needs to do that him/herself using a text editor (which is very easy and works very well). The behaviour should consistently be: Use the default config (in home or /tmp) or use the config specified in the environment variable.

Do you perhaps know what Path.home() returns in an apptainer that runs without $HOME?

marcelzwiers commented 9 months ago

Do you perhaps know what Path.home() returns in an apptainer that runs without $HOME?

Never mind, I found the answer. The problem I have is that tempfile.gettempdir() doesn't necessarily return a usable result...

marcelzwiers commented 9 months ago

If your container has a tempdir (doesn't it?) then I hope my last commit works as expected...

marcelzwiers commented 9 months ago

I just tested it in my own apptainer and there things now work (latest version from github) without setting any environment variable. So I guess that settles it, or?