Can path to .bidscoin be configured?

[mateuszpawlik]

General summary I've just realized that the newer BIDScoin hard-codes the path to .bidscoin directory to reside in home. What if I don't want it there?

Additional detail I came across that when executing BIDScoin (dicomsort tool) in a container without home mounted.

Next steps It's not up to me. Maybe at least an option to configure that? If not, make it more visible in the documentation?

[marcelzwiers]

apptainer run --no-home bidscoin.sif

```console INFO: fuse2fs not found, will not be able to mount EXT3 filesystems Creating BIDScoin configuration: -> /tmp/.bidscoin/4.3.0/config.toml -> /tmp/.bidscoin/4.3.0/templates/bidsmap_bids2bids.yaml -> /tmp/.bidscoin/4.3.0/templates/bidsmap_dccn.yaml -> /tmp/.bidscoin/4.3.0/templates/bidsmap_sst.yaml -> /tmp/.bidscoin/4.3.0/templates/schema.json usage: bidscoin [-h] [-l] [-p] [-i INSTALL [INSTALL ...]] [-u UNINSTALL [UNINSTALL ...]] [-d DOWNLOAD] [-t [TEST]] [-b BIDSMAPTEST] [-c CREDITS [CREDITS ...]] [--tracking {yes,no,show}] [-v] BIDScoin is a toolkit to convert raw data-sets according to the Brain Imaging Data Structure (BIDS) The basic workflow is to run these two tools: $ bidsmapper sourcefolder bidsfolder # This produces a study bidsmap and launches a GUI $ bidscoiner sourcefolder bidsfolder # This converts your data to BIDS according to the study bidsmap Default settings and template bidsmaps are stored in the `.bidscoin` folder in your home directory (you can modify them to your needs with any plain text editor) Set the environment variable `BIDSCOIN_DEBUG=TRUE` to run BIDScoin in a more verbose logging mode and `BIDSCOIN_CONFIG=/writable/path/to/config.toml` if you like to use a different config file. Citation reports can be generated with the help of duecredit (https://github.com/duecredit/duecredit) For more documentation see: https://bidscoin.readthedocs.io options: -h, --help show this help message and exit -l, --list List all executables (i.e. the apps, bidsapps and utilities) -p, --plugins List all installed plugins and template bidsmaps -i INSTALL [INSTALL ...], --install INSTALL [INSTALL ...] A list of template bidsmaps and/or bidscoin plugins to install -u UNINSTALL [UNINSTALL ...], --uninstall UNINSTALL [UNINSTALL ...] A list of template bidsmaps and/or bidscoin plugins to uninstall -d DOWNLOAD, --download DOWNLOAD Download tutorial MRI data to the DOWNLOAD folder -t [TEST], --test [TEST] Test the bidscoin installation and template bidsmap -b BIDSMAPTEST, --bidsmaptest BIDSMAPTEST Test the run-items and their bidsnames of all normal runs in the study bidsmap. Provide the bids-folder or the bidsmap filepath -c CREDITS [CREDITS ...], --credits CREDITS [CREDITS ...] Show duecredit citations for your BIDS repository. You can also add duecredit summary arguments (without dashes), e.g. `style {apa,harvard1}` or `format {text,bibtex}`. --tracking {yes,no,show} Show the usage tracking info {show}, or set usage tracking to {yes} or {no} -v, --version Show the installed version and check for updates examples: bidscoin -l bidscoin -d data/bidscoin_tutorial bidscoin -t bidscoin -t my_template_bidsmap bidscoin -b my_study_bidsmap bidscoin -i data/my_template_bidsmap.yaml downloads/my_plugin.py bidscoin -c myproject/bids bidscoin -c myproject/bids format bibtex bidscoin --tracking show ```

apptainer run bidscoin.sif

```console INFO: fuse2fs not found, will not be able to mount EXT3 filesystems Creating BIDScoin configuration: -> /home/marzwi/.bidscoin/4.3.0/config.toml -> /home/marzwi/.bidscoin/4.3.0/templates/bidsmap_bids2bids.yaml -> /home/marzwi/.bidscoin/4.3.0/templates/bidsmap_dccn.yaml -> /home/marzwi/.bidscoin/4.3.0/templates/bidsmap_sst.yaml -> /home/marzwi/.bidscoin/4.3.0/templates/schema.json usage: bidscoin [-h] [-l] [-p] [-i INSTALL [INSTALL ...]] [-u UNINSTALL [UNINSTALL ...]] [-d DOWNLOAD] [-t [TEST]] [-b BIDSMAPTEST] [-c CREDITS [CREDITS ...]] [--tracking {yes,no,show}] [-v] BIDScoin is a toolkit to convert raw data-sets according to the Brain Imaging Data Structure (BIDS) The basic workflow is to run these two tools: $ bidsmapper sourcefolder bidsfolder # This produces a study bidsmap and launches a GUI $ bidscoiner sourcefolder bidsfolder # This converts your data to BIDS according to the study bidsmap Default settings and template bidsmaps are stored in the `.bidscoin` folder in your home directory (you can modify them to your needs with any plain text editor) Set the environment variable `BIDSCOIN_DEBUG=TRUE` to run BIDScoin in a more verbose logging mode and `BIDSCOIN_CONFIG=/writable/path/to/config.toml` if you like to use a different config file. Citation reports can be generated with the help of duecredit (https://github.com/duecredit/duecredit) For more documentation see: https://bidscoin.readthedocs.io options: -h, --help show this help message and exit -l, --list List all executables (i.e. the apps, bidsapps and utilities) -p, --plugins List all installed plugins and template bidsmaps -i INSTALL [INSTALL ...], --install INSTALL [INSTALL ...] A list of template bidsmaps and/or bidscoin plugins to install -u UNINSTALL [UNINSTALL ...], --uninstall UNINSTALL [UNINSTALL ...] A list of template bidsmaps and/or bidscoin plugins to uninstall -d DOWNLOAD, --download DOWNLOAD Download tutorial MRI data to the DOWNLOAD folder -t [TEST], --test [TEST] Test the bidscoin installation and template bidsmap -b BIDSMAPTEST, --bidsmaptest BIDSMAPTEST Test the run-items and their bidsnames of all normal runs in the study bidsmap. Provide the bids-folder or the bidsmap filepath -c CREDITS [CREDITS ...], --credits CREDITS [CREDITS ...] Show duecredit citations for your BIDS repository. You can also add duecredit summary arguments (without dashes), e.g. `style {apa,harvard1}` or `format {text,bibtex}`. --tracking {yes,no,show} Show the usage tracking info {show}, or set usage tracking to {yes} or {no} -v, --version Show the installed version and check for updates examples: bidscoin -l bidscoin -d data/bidscoin_tutorial bidscoin -t bidscoin -t my_template_bidsmap bidscoin -b my_study_bidsmap bidscoin -i data/my_template_bidsmap.yaml downloads/my_plugin.py bidscoin -c myproject/bids bidscoin -c myproject/bids format bibtex bidscoin --tracking show ```

apptainer run --env BIDSCOIN_CONFIG=~/somefolder/config.toml bidscoin.sif

```console INFO: fuse2fs not found, will not be able to mount EXT3 filesystems Creating BIDScoin configuration: -> /home/marzwi/somefolder/config.toml -> /home/marzwi/somefolder/templates/bidsmap_bids2bids.yaml -> /home/marzwi/somefolder/templates/bidsmap_dccn.yaml -> /home/marzwi/somefolder/templates/bidsmap_sst.yaml -> /home/marzwi/somefolder/templates/schema.json usage: bidscoin [-h] [-l] [-p] [-i INSTALL [INSTALL ...]] [-u UNINSTALL [UNINSTALL ...]] [-d DOWNLOAD] [-t [TEST]] [-b BIDSMAPTEST] [-c CREDITS [CREDITS ...]] [--tracking {yes,no,show}] [-v] BIDScoin is a toolkit to convert raw data-sets according to the Brain Imaging Data Structure (BIDS) The basic workflow is to run these two tools: $ bidsmapper sourcefolder bidsfolder # This produces a study bidsmap and launches a GUI $ bidscoiner sourcefolder bidsfolder # This converts your data to BIDS according to the study bidsmap Default settings and template bidsmaps are stored in the `.bidscoin` folder in your home directory (you can modify them to your needs with any plain text editor) Set the environment variable `BIDSCOIN_DEBUG=TRUE` to run BIDScoin in a more verbose logging mode and `BIDSCOIN_CONFIG=/writable/path/to/config.toml` if you like to use a different config file. Citation reports can be generated with the help of duecredit (https://github.com/duecredit/duecredit) For more documentation see: https://bidscoin.readthedocs.io options: -h, --help show this help message and exit -l, --list List all executables (i.e. the apps, bidsapps and utilities) -p, --plugins List all installed plugins and template bidsmaps -i INSTALL [INSTALL ...], --install INSTALL [INSTALL ...] A list of template bidsmaps and/or bidscoin plugins to install -u UNINSTALL [UNINSTALL ...], --uninstall UNINSTALL [UNINSTALL ...] A list of template bidsmaps and/or bidscoin plugins to uninstall -d DOWNLOAD, --download DOWNLOAD Download tutorial MRI data to the DOWNLOAD folder -t [TEST], --test [TEST] Test the bidscoin installation and template bidsmap -b BIDSMAPTEST, --bidsmaptest BIDSMAPTEST Test the run-items and their bidsnames of all normal runs in the study bidsmap. Provide the bids-folder or the bidsmap filepath -c CREDITS [CREDITS ...], --credits CREDITS [CREDITS ...] Show duecredit citations for your BIDS repository. You can also add duecredit summary arguments (without dashes), e.g. `style {apa,harvard1}` or `format {text,bibtex}`. --tracking {yes,no,show} Show the usage tracking info {show}, or set usage tracking to {yes} or {no} -v, --version Show the installed version and check for updates examples: bidscoin -l bidscoin -d data/bidscoin_tutorial bidscoin -t bidscoin -t my_template_bidsmap bidscoin -b my_study_bidsmap bidscoin -i data/my_template_bidsmap.yaml downloads/my_plugin.py bidscoin -c myproject/bids bidscoin -c myproject/bids format bibtex bidscoin --tracking show ```

[mateuszpawlik]

Do you perhaps know what Path.home() returns in an apptainer that runs without $HOME?

Never mind, I found the answer. The problem I have is that tempfile.gettempdir() doesn't necessarily return a usable result...

Yes, I know. It points to host HOME. I will cite the relevant issue again ;-) https://github.com/apptainer/apptainer/issues/388

[mateuszpawlik]

If your container has a tempdir (doesn't it?) then I hope my last commit works as expected...

It does. But depending on the execution parameters, it may be mounted from the host or not.

[marcelzwiers]

As long as tempfile.gettempdir() returns a result then things are ok. BIDScoin needs some space to write temporary stuff (also for unpacking zipped data, etc)

[marcelzwiers]

Thanks for reporting this, I will try to release a new version today (I think the best and most robust version so far)

[mateuszpawlik]

Inside of the container

version 4.2.1

I fix it with --scratch $HOME when Apptainer is executed with --no-home, because bidscoin keeps looking up $HOME although it isn't mounted.

Ok, that's not good, I will make a fallback-fix for this (in $TMPDIR?)

I like it.

current master version (cd409e1)

Adding --env BIDSCOIN_CONFIG=/root/.bidscoin/4.3.0/config.toml to Apptainer command should help, because bidscoin doesn't know where is .bidscoin without BIDSCOIN_CONFIG set.

So you pass a config file, that should work, but pointing it to /root puzzles me.

When you build an Apptainer image (especially from a Docker image), there is only root user, and therefore there are no user homes. During building of the container, HOME points to /root. In order to reuse the files created by BIDScoin's installation, I point to where they were created.

If it is not set, bidscoin expects .bidscoin in HOME, which for installation in Apptainer is not true (HOME always points to host home).

If it's not set, then bidscoin will create it in home, but I'll make that fallback-fix that should cover this

Ok. This seems to be on Apptainer and --no-home flag. HOME points to home in the host but --no-home doesn't mount it.

If config.toml needs to be overwritten, it can be mounted additionally (--bind /path/to/my/config.toml:/root/.bidscoin/4.3.0/config.toml). If BIDSCOIN_CONFIG is set to something else than /root/.bidscoin/4.3.0/config.toml, the parent directory has to be mounted.

Sorry, I'm not that used to working with containers, but I don't see why you have to bind that root path? Isn't passing --env BIDSCOIN_CONFIG=/some/mounted/folder/bidscoin/config.toml sufficient?

In this example (second sentence of my comment), you still have to mount --bind /some/mounted/folder/bidscoin.

In general

I should not need to set BIDSCOIN_CONFIG and care about the .bidscoin directory, especially if I use bidsmapper with --template option. The config file only sets this template file and the tracking.

I agree that the config folder shouldn't be a user concern and I'm surely open for suggestions to improve things here.

It may be solved with your newest changes. I'll test it in a bit.

Usually, I can set the options in a config file, or provide them with CLI, with some priorities.

Well, that sounds great but on the other hand, it is also very common for applications to simply keep their settings in a hidden file or directory in home (e.g. dcm2niix).

Agree. It is fine if BIDScoin creates itself a directory. Though, in my opinion it should create it where it's installed. If I install it in the container, .bindcoin should be in the container and not recreate it in the host. No?

Bidscoin could remember where it created .bidscoin directory, whether BIDSCOIN_CONFIG is set or not.

I don't see how bidscoin can remember that without storing it somewhere on disk. Which is what the config file is

Good point. Maybe the problem is really only this Apptainer issue.

Then, it could preset the options from the config file in there, and optionally overwrite them with values provided with CLI. This would make the behavior consistent.

Currently I don't provide a tool or CLI options for editing/overruling the config file, the user needs to do that him/herself using a text editor (which is very easy and works very well). The behaviour should consistently be: Use the default config (in home or /tmp) or use the config specified in the environment variable.

I like the tmp solution. In my particular case, however, it won't work. I mount a subdirectory of my home in the container (absolute path). It means that Path.home().exists() returns true in the container, but it is not writable. One could use os.access('my_folder', os.W_OK) to additionally check permission. But I really don't want to complicate it anymore :sob:

>>> Path.home().exists()
True
>>> os.access(Path.home(), os.W_OK)
False

[mateuszpawlik]

Many thanks for dealing with that, although it wasn't on you. I learned a lot.

[marcelzwiers]

Agree. It is fine if BIDScoin creates itself a directory. Though, in my opinion it should create it where it's installed. If I install it in the container, .bindcoin should be in the container and not recreate it in the host. No?

No, because then it is not writable

It means that Path.home().exists() returns true in the container

I didn't know that, I'll add an even more thorough test then

I like the tmp solution. In my particular case, however, it won't work

You mean in your case tempfile.gettempdir() does not return a writable tmpdir? If so, then I am afraid I cannot fix that

But I really don't want to complicate it anymore

No, I really appreciate it, it makes bidscoin more robust

[marcelzwiers]

One could use os.access('my_folder', os.W_OK) to additionally check permission.

Done!

[mateuszpawlik]

Agree. It is fine if BIDScoin creates itself a directory. Though, in my opinion it should create it where it's installed. If I install it in the container, .bindcoin should be in the container and not recreate it in the host. No?

No, because then it is not writable

Right :see_no_evil:

I like the tmp solution. In my particular case, however, it won't work

You mean in your case tempfile.gettempdir() does not return a writable tmpdir? If so, then I am afraid I cannot fix that

/tmp is fine:

>>> Path(tempfile.gettempdir())
PosixPath('/tmp')
>>> os.access(Path(tempfile.gettempdir()), os.W_OK)
True

I meant that it won't work for me if you only check for existence of HOME but not if it's writable. If you check both and resort on fail to tmp, it should work for me too.

But I really don't want to complicate it anymore

No, I really appreciate it, it makes bidscoin more robust

:-)

[marcelzwiers]

FYI, I changed BIDSCOIN_CONFIG to BIDSCOIN_CONFIGDIR to better reflect what happens / avoid config conflicts. Another way of seeing it is that the templates are also config files, not just config.toml

[mateuszpawlik]

That is actually great :-) I agree. Thanks.