Farhad-Heybati
commented
3 years ago I think the issue is coming from the fact that the output of preprocessing is defined by PipelineData: output_data = PipelineData(name="train{}".format(str(i)), datastore=ds, output_path_on_compute='mnli_data/processed_train/batch{}.csv'.format(str(i))) By default, the path to the data file is based on an environment variable: $AZUREML_DATAREFERENCE for example: processed_train_files[0] => $AZUREML_DATAREFERENCE_train0 When I print the value of the file path in preprocessing step and in the training step, these are differents: preprocess step: /mnt/batch/tasks/shared/LS_root/jobs/gaia-ml-wks/azureml/273296fe-21e3-44fa-84d1-6a6b33c04031/mounts/workspaceblobstore/azureml/273296fe-21e3-44fa-84d1-6a6b33c04031/train0' Bert training step: /mnt/batch/tasks/shared/LS_root/jobs/gaia-ml-wks/azureml/51dd8b67-13a7-40a1-a9ff-eef158a300c5/mounts/workspaceblobstore/azureml/273296fe-21e3-44fa-84d1-6a6b33c04031/train0
The $AZUREML_DATAREFERENCE value change based on the runID
Description
tc_bert_azureml.ipynb Error message: wc: /mnt/batch/tasks/shared/LS_root/jobs/gaia-ml-wks/azureml/3576038c-f399-4522-8674-383ad7cd316b/mounts/workspaceblobstore/azureml/88f85fcf-eaf4-4d58-bff2-712b35cb50aa/train0,: No such file or directory
How do we replicate the bug?
Just run the notebook tc_bert_azureml
Expected behavior (i.e. solution)
Other Comments