BUG: recycle mixup between default cache and output `cache:key`

[ebolyen]

Bug Description It appears that when an output is set via some-cache:key the cache variable shadows the recycle cache, independent of --use-cache

Fortunately, this cleanup happens after saving the outputs, so other than the exit code being nonzero, so it's not a huge deal.

Saved SampleData[Kraken2Report] to: kraken_report.qza
Added SampleData[Kraken2Output] to cache: cache as: kraken_out
Added FeatureTable[Frequency] to cache: cache as: bracken_table
Traceback (most recent call last):
  File "/home/evan/.conda/envs/q2-dev/lib/python3.8/site-packages/qiime2/core/cache.py", line 1004, in remove
    os.remove(self.keys / key)
FileNotFoundError: [Errno 2] No such file or directory: 'cache/keys/recycle_moshpit_classify_kraken_bracken_4a28b246fa005793fb37b8719c4f0811a5ca64ca'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/evan/.conda/envs/q2-dev/bin/qiime", line 33, in <module>
    sys.exit(load_entry_point('q2cli', 'console_scripts', 'qiime')())
  File "/home/evan/.conda/envs/q2-dev/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/evan/.conda/envs/q2-dev/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/evan/.conda/envs/q2-dev/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/evan/.conda/envs/q2-dev/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/evan/.conda/envs/q2-dev/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/evan/.conda/envs/q2-dev/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/evan/consulting/qiime2/q2cli/q2cli/commands.py", line 534, in __call__
    cache.remove(recycle_pool)
  File "/home/evan/.conda/envs/q2-dev/lib/python3.8/site-packages/qiime2/core/cache.py", line 1006, in remove
    raise KeyError(f"The cache '{self.path}' does not contain the"
KeyError: "The cache 'cache' does not contain the key 'recycle_moshpit_classify_kraken_bracken_4a28b246fa005793fb37b8719c4f0811a5ca64ca'"

[ebolyen]

We may also want to consider what happens when two different actions are running at the same time with a valid recycle pool. One will win the race to delete, and the other will be sad.

[Oddant1]

Good point, wondering what the most sane approach to take here is. We can probably use the lock to create some kind of counter tracking how many actions are using the pool. Like a semaphore where the exiting process only removes the recycle pool if, after it decrements the "semaphore" for itself the value is 0.

[Oddant1]

We could and probably should also lock the entire index checking process. That may actually solve this simpler

[Oddant1]

Yeah because then if we find stuff in the recycle pool we will copy it into the process pool then we can use the versions of the artifacts backed by the process pool.

[Oddant1]

@ebolyen I'm pretty sure we closed this but actually closing it fell through because it was a bit sporadic