Exception: Could not find import model store for directory

[bgruening]

We have been getting a lot of error reports like:

Traceback (most recent call last):
  File "/opt/galaxy/server/lib/galaxy/jobs/runners/__init__.py", line 628, in _finish_or_resubmit_job
    job_wrapper.finish(
  File "/opt/galaxy/server/lib/galaxy/jobs/__init__.py", line 1802, in finish
    import_model_store = store.get_import_model_store_for_directory(
  File "/opt/galaxy/server/lib/galaxy/model/store/__init__.py", line 1222, in get_import_model_store_for_directory
    raise Exception(
Exception: Could not find import model store for directory [/data/jwd/main/051/948/51948333/metadata/outputs_populated] (full path [/data/jwd/main/051/948/51948333/metadata/outputs_populated])

This seems to be true, but it was not obvious why this has happen

galaxy@sn06:~$ ll /data/jwd/main/051/948/51948333/metadata/
total 148
-rw-r--r-- 1 galaxy galaxy     20 Nov  5 08:53 metadata_kwds_tab
-rw-r--r-- 1 galaxy galaxy      0 Nov  5 08:53 metadata_out_tab
-rw-r--r-- 1 galaxy galaxy      2 Nov  5 08:53 metadata_override_tab
-rw-r--r-- 1 galaxy galaxy     41 Nov  5 08:53 metadata_results_tab
-rw-r--r-- 1 galaxy galaxy  10775 Nov  5 08:53 object_store_conf.json
drwxr-xr-x 2 galaxy galaxy   4096 Nov  5 08:53 outputs_new
-rw-r--r-- 1 galaxy galaxy   1910 Nov  5 08:53 params.json
-rw-r--r-- 1 galaxy galaxy 120181 Nov  5 08:53 registry.xml
-rw-r--r-- 1 galaxy galaxy     73 Nov  5 08:53 set.py

At first, that seems like a problem with metadata_strategy: 'extended' which we have enabled by default. Disabling this revealed a problem with setting metadata in general, which is handled by celery in our setup.

It turned out that a recent Galaxy update has introduced a new dependency that was not synced to the external celery node. So celery was the problem behind this error. Fixing celery and refreshing the venv of celery fixed it.

Not sure if we can improve the initial error message, to direct admins directly to the set_metadata process.

[bgruening]

I created this issue to help others that might see similar errors. Nothing really actionable to do here I think, besides maybe improving the error message.

[mvdbeek]

That's definitely something to work on though!

[bgruening]

We do see this error again. Not sure what we can do this time.

This is also "funny":

I'm not sure what is wrong with celery3 but I restarted it several times and it does not get better.

[bgruening]

I will stop that task handler now and see if we still get those error reports.

[bgruening]

@gallardoalba for your information.

[mira-miracoli]

Now we have many failed jobs on almost all workers, and small (200kb) upload jobs are not finished by celery.