SwissDataScienceCenter / renku-python

A Python library for the Renku collaborative data science platform.
https://renku-python.readthedocs.io/
Apache License 2.0
37 stars 29 forks source link

Renku update failure #1107

Closed m-alisafaee closed 3 years ago

m-alisafaee commented 4 years ago

To reproduce:

$ renku dataset create mydata
$ renku dataset add mydata -s csse_covid_19_data/csse_covid_19_time_series -d csv https://github.com/CSSEGISandData/COVID-19.git
$ renku rm data/mydata/csv/.gitignore data/mydata/csv/README.md
$ mkdir data/mydata/preprocessed
$ echo "import shutil
from pathlib import Path

for file_ in Path('data/mydata/csv/').glob('*csv'):
    output = Path('data/mydata/preprocessed/') / file_.name
    shutil.copy(file_, output)
" > notebooks/script.py
$ git add notebooks
$ git commit -m 'add script'
$ renku run python3 notebooks/script.py

# update the script

$ echo '# some comment' >> notebooks/script.py
$ git commit -am 'update script'
$ renku update

Resolved '.renku/workflow/e3aa8f328d2f458f9009f41d39cf5877.cwl' to 'file:///home/mohammad/playground/tmp/.renku/workflow/e3aa8f328d2f458f9009f41d39cf5877.cwl'
[workflow ] start
[workflow ] starting step step_1
[step step_1] start
[job step_1] /tmp/m42qnny4$ python3 \
    /tmp/m42qnny4/notebooks/script.py
[job step_1] Job error:
("Error collecting output for parameter 'output_0':\n.renku/workflow/0d30b1c260004c818d55b98c4b1b71e5_python3.cwl:21:7: Did not find output file with glob pattern: '['data/mydata/preprocessed/time_series_19-covid-Recovered.csv']'", {})
[job step_1] completed permanentFail
[step step_1] Output is missing expected field file:///home/mohammad/playground/tmp/.renku/workflow/e3aa8f328d2f458f9009f41d39cf5877.cwl#step_1/output_1
[step step_1] Output is missing expected field file:///home/mohammad/playground/tmp/.renku/workflow/e3aa8f328d2f458f9009f41d39cf5877.cwl#step_1/output_2
[step step_1] Output is missing expected field file:///home/mohammad/playground/tmp/.renku/workflow/e3aa8f328d2f458f9009f41d39cf5877.cwl#step_1/output_0
[step step_1] completed permanentFail
[workflow ] completed permanentFail
Error: Unable to finish re-executing workflow; check the workflow execution outline above and the generated /home/mohammad/playground/tmp/.renku/workflow/e3aa8f328d2f458f9009f41d39cf5877.cwl file for potential issues, then remove the /home/mohammad/playground/tmp/.renku/workflow/e3aa8f328d2f458f9009f41d39cf5877.cwl file and try again
Panaetius commented 3 years ago

This bug does not happen anymore. The update will fail anyways, because you didn't specify data/mydata/csv/ as an explicit input, so it doesn't get copied by cwl and no output is produced.

Closing this as the issue itself is solved.