Dagstermill: context.solid_config is not defined in the notebook

[lee335]

Hi, we use dagstermill (0.7.16) to trigger the training of our machine learning model on aws sagemaker.

Currently we encounter an issue, where the training pipeline can be started using the playground on dagit. But whenever the dagstermill part is reached, the pipeline fails because the context.solid_config is not defined in the notebook, although everything for the notebook solid was configured correctly on the playground. However restarting the pipeline several times without changing anything, the dagstermill part will eventually get executed correctly and the pipeline finishes successfully.

Can you help us with this issue?

[tinkerware]

We have seen this happen as well (we were on 0.6 at the time), but did not have much time to debug it.

[lee335]

We also see this issue when the pipeline is getting executed using the cron scheduler. Since we want this pipeline to be triggered twice a day in our production pipeline, is there any workaround for this problem?

[lee335]

We have found a workaround that works for us. So if any one runs into the same problem, they can try the following approach:

• create a new normal solid
• in this new solid read the config parameters meant for the notebook and make it as an output available
• in the notebook solid don't read the config parameters but instead define an input variable
• this input variable will then have the notebook config parameters when connected to the new solid

[jeremyadamsfisher]

This is still an issue with the context.op_config as well