Have a discussion on the issue and potential solution.
Description
add by @jusongyu
Calculations which pass (part of) their outputs on as a RemoteData input are broken once their scratch directory is cleaned. It becomes problematic however when caching is introduced: When trying to re-run the calculation (to regenerate the remote folder), caching will happily use the existing calculation (without a remote folder). As a result, subsequent calculations which use the remote folder will fail. Moreover, the cached Calculation node point to the same remote folder which is a 'shallow copy' of the original calculation node that causes the issue when cleaning the remote workdir of cached node, the original node is unable to used for parent calculation to pass their outputs on as a RemoteData. The caching policy needs to be redesigned a bit to overcome these issues and balance with not increasing the connection burden and remote disk storage too much.
sphuber
commented
1 year ago
Next action
Have a discussion on the issue and potential solution.
Description
Calculations which pass (part of) their outputs on as a RemoteData input are broken once their scratch directory is cleaned. It becomes problematic however when caching is introduced: When trying to re-run the calculation (to regenerate the remote folder), caching will happily use the existing calculation (without a remote folder). As a result, subsequent calculations which use the remote folder will fail. Moreover, the cached Calculation node point to the same remote folder which is a 'shallow copy' of the original calculation node that causes the issue when cleaning the remote workdir of cached node, the original node is unable to used for parent calculation to pass their outputs on as a RemoteData. The caching policy needs to be redesigned a bit to overcome these issues and balance with not increasing the connection burden and remote disk storage too much.
See https://github.com/aiidateam/aiida-core/issues/3735 and https://github.com/aiidateam/aiida-core/issues/5178 for more details.
AEP open at https://github.com/aiidateam/AEP/pull/35