Premade BIDS Database Error

smeisler commented 2 years ago

Environment

fitlins Singularity container version 0.10.1
PyBids version 0.14.0
CentOs7

Expected Behavior

fitlins to begin running

Observed Behavior

Error (log below)

Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such column: tags.is_metadata

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/bin/fitlins", line 8, in <module>
    sys.exit(main())
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/fitlins/cli/run.py", line 432, in main
    sys.exit(run_fitlins(sys.argv[1:]))
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/fitlins/cli/run.py", line 356, in run_fitlins
    subject_list = fub.collect_participants(layout, participant_label=opts.participant_label)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/fitlins/utils/bids.py", line 91, in collect_participants
    all_participants = sorted(layout.get_subjects())
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/bids/layout/layout.py", line 673, in get
    metadata = target not in self.get_entities(metadata=False)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/bids/layout/layout.py", line 374, in get_entities
    results = query.all()
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3373, in all
    return list(self)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
    return self._execute_and_instances(context)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
    ret = self._execute_context(
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such column: tags.is_metadata
[SQL: SELECT entities.name AS entities_name, entities.mandatory AS entities_mandatory, entities.pattern AS entities_pattern, entities.directory AS entities_directory, entities._dtype AS entities__dtype 
FROM entities JOIN tags ON entities.name = tags.entity_name 
WHERE tags.is_metadata = 0]
(Background on this error at: http://sqlalche.me/e/13/e3q8)

Steps to Reproduce

Create PyBids environment: pybids layout --index-metadata --reset-db $bids $db_dir (PyBids 0.14.0)
Run FitLins: singularity run -e -B ${scratch},${bids_dir} $IMG ${bids_dir} ${output_dir} run -m ${bids_dir}/models/phono-model.json -d ${output_dir}/fmriprep -w $scratch --database-path ${bids_dir}/code/pybids_db --participant-label ${subject:4} --space MNI152NLin2009cAsym -s 5

adelavega commented 2 years ago

I wonder if the database is made w/ an distinct version of fitlins/pybids or the arguments are not identical upon creation.

I'm guessing you are trying to make the db "offline" (i.e. not in singularity) to save compute? If you were to let FitLins create the db that would be more likely that it could re-use it (as a workaround).

Regardless what you're trying to do should work in principle.

smeisler commented 2 years ago

Something that may also relate to this is that I made my db before running fmriprep, and I believe FitLin indexes the fmriprep directory. Should the db passed into FitLins index fmriprep too?

adelavega commented 2 years ago

Yes, it should. That is what I meant in terms of "identical" parameters. I doubt that's what leads to this error though, but it's possible.

The code FitLins uses to create a BIDSLayout looks like:


    indexer = bids.BIDSLayoutIndexer(ignore=ignore, force_index=force_index)
    layout = bids.BIDSLayout(
        opts.bids_dir,
        derivatives=derivatives,
        database_path=database_path,
        reset_database=reset_database,
        indexer=indexer,
    )

so actually I don't see much difference except the derivatives (the rest of the arguments, such as force_index come from teh CLI arguments.

Try to create a layout that indexes derivatives as well and see if you can load it w/ FitLins.

Also, FitLins currently uses pybids v0.15, so that's another possible divergence.

poldracklab / fitlins