riga / law

Build large-scale task workflows: luigi + job submission + remote targets + environment sandboxing using Docker/Singularity
http://law.readthedocs.io
BSD 3-Clause "New" or "Revised" License
96 stars 39 forks source link

Fix interference between input localization decorator and sandbox stage-in. #168

Closed riga closed 10 months ago

riga commented 11 months ago

When using both the sandbox stage-in mechanism and input target localization (e.g. through the law.decorator.localize() decorator), the localized targets are not existing.

It seems that, when localization is used, the stage-in is happening twice: once has part of the localization and once when task.input() is actually called. This leads to undesired side effects.

The reason is that the localize() decorator simply patches the input and output attributes (both methods) for the duration of the run() call, whereas the stage-in mechanism must hook __getattribute__ to provide the changed methods.

As a result, input() a calls _staged_input(), which is meant to call the original inputs() method. The latter is, however, the localized one such that the stage-in mechanism considers the localized targets as the original ones which is a bug.

The stage-in should always interact the with unchanged, original targets, and in this case, localization should deal with what the stage-in provides.