adrtsc
commented
8 months ago Hi @jluethi,
I already talked to you about this, but this issue was not fixed yet for me. For reference, here again the behaviour I observed:
- application of registration worked fine for few wells of a plate or single wells
- the task however failed when trying to process a whole 384 well plate
- the task failed always for the same wells, but running these wells separately worked fine
When the task failed, I got this error message:
Traceback (most recent call last):
File "/net/nfs4/pelkmanslab-fileserver-fractal/data/homes/fractal/20230627_joel_fractal/fractal-demos/examples/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.14.0/venv/lib/python3.10/site-packages/fractal_tasks_core/tasks/apply_registration_to_image.py", line 213, in apply_registration_to_image
old_table_group = zarr.open_group(table_dict[table], mode="r")
File "/data/homes/fractal/20230627_joel_fractal/fractal-demos/examples/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.14.0/venv/lib/python3.10/site-packages/zarr/hierarchy.py", line 1532, in open_group
raise GroupNotFoundError(path)
zarr.errors.GroupNotFoundError: group not found at path ''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/net/nfs4/pelkmanslab-fileserver-fractal/data/homes/fractal/20230627_joel_fractal/fractal-demos/examples/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.14.0/venv/lib/python3.10/site-packages/fractal_tasks_core/tasks/apply_registration_to_image.py", line 377, in <module>
run_fractal_task(
File "/data/homes/fractal/20230627_joel_fractal/fractal-demos/examples/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.14.0/venv/lib/python3.10/site-packages/fractal_tasks_core/tasks/_utils.py", line 79, in run_fractal_task
metadata_update = task_function(**pars)
File "pydantic/decorator.py", line 40, in pydantic.decorator.validate_arguments.validate.wrapper_function
File "pydantic/decorator.py", line 134, in pydantic.decorator.ValidatedFunction.call
File "pydantic/decorator.py", line 206, in pydantic.decorator.ValidatedFunction.execute
File "/net/nfs4/pelkmanslab-fileserver-fractal/data/homes/fractal/20230627_joel_fractal/fractal-demos/examples/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.14.0/venv/lib/python3.10/site-packages/fractal_tasks_core/tasks/apply_registration_to_image.py", line 216, in apply_registration_to_image
old_table_group = zarr.open_group(table_dict[table], mode="r")
File "/data/homes/fractal/20230627_joel_fractal/fractal-demos/examples/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.14.0/venv/lib/python3.10/site-packages/zarr/hierarchy.py", line 1532, in open_group
raise GroupNotFoundError(path)
zarr.errors.GroupNotFoundError: group not found at path ''My workaround at the moment is to put the group loading/sleep code section into a while loop. This task then runs through and I haven't had any issues with it so far.
for table in table_dict.keys():
logger.info(f"Copying table: {table}")
# Get the relevant metadata of the Zarr table & add it
# See issue #516 for the need for this workaround
checkpoint = 0
while checkpoint == 0:
try:
old_table_group = zarr.open_group(table_dict[table], mode="r")
checkpoint += 1
except zarr.errors.GroupNotFoundError:
logger.warning(
f"Table {table} not found yet. Waiting 5 seconds "
"before trying again"
)
time.sleep(5)
jluethi
The Apply Registration to image task can hit a race condition. Because we copy the standard ROI tables from the reference cycle, but also modify the reference cycle, the following happens:
This only happens in the
overwrite_input = Truescenario. Because as part of that, we create a new Zarr group (e.g. 0_registered). We then remove the old zarr group (0) and rename the registered zarr group to 0.While the task is running for cycle 0 and is in the process of removing the old zarr group, the task running for a different cycle may want to read the tables from the reference cycle. But that table temporarily doesn't exist. This results in the following error:
I'm creating a PR to make this race condition way less likely:
group_name_tmp(e.g. (0_tmp). Then rename the new group to the old group name (e.g. from 0_registered to 0). Both those renamings are very fast. Only after that, remove the tmp folder.This should make it very unlikely to hit the issue. But the real solution would be to remove the need to load something from the reference cycle in a task that also runs in parallel on said reference cycle (basically, find a way to generate the new, correct ROI tables from the cycle itself).