Pipeline editor does not refresh runtime image list after updates via the metadata editor

[ptitzler]

1.1

Scenario:

• I had previously created a runtime image configuration. (days ago)

• In this configuration I changed today the Docker image name to something else.

• I created a new pipeline and added a notebook.

• I assigned the updated runtime image configuration as "runtime image" for the notebook node.

• I saved the pipeline.

• The pipeline file includes a reference to the old Docker image.

  "pipelines": [
    {
      "id": "c26a08f5-6a74-4edc-9e74-f6fd7dcfb2b7",
      "nodes": [
        {
          "id": "b5a7da08-35c6-44a4-afe2-6067bfabe181",
          "type": "execution_node",
          "op": "execute-notebook-node",
          "app_data": {
            "filename": "Untitled.ipynb",
            "runtime_image": "ptitzler/my-runtime-image",
            "env_vars": [],
            "include_subdirectories": false,

[ptitzler]

It appears that the runtime image list in the node properties view is not always refreshed, causing a discrepancy between what's cached and what's stored in the metadata.

[ptitzler]

We also recreated the issue using the following scenario in 1.2:

• create a notebook
• create a pipeline
• create a runtime image
• add notebook to pipeline
• open notebook properties -> the new runtime image is not listed in the list

Edit 1: the runtime image shows up after JL was restarted. Looks like the pipeline editor is using cached information.

[kevin-bates]

I believe you're describing two different issues here.

1. The opening description describes a scenario where a pipeline node's reference to an image did not reflect changes to the image-name that took place in the runtime-image instance. If I'm understanding that description, the image name (subtly) changed from ptitzler/my-runtime-image to ptitzler/my-runtime-image:latest (i.e., the 'latest' tag was appended). This is working as designed since it was decided that we should err on the side of reproducibility and not potentially side-affect existing pipelines via the update of an indirect reference.

2. The latest comment describes an issue more about caching, which is indeed used on the client and should probably be more closely examined on how/when to refresh. If the runtime image was created via the UI, then we should certainly refresh the cache. Its tougher to know when to refresh if the change takes place on the server. It might be best, for now, to err on the side of refresh when entering the properties dialog.

[ptitzler]

... two different issues ...

It's actually the same scenario and cause. Let me try to clarify number 1

• create a runtime image config
• restart JL
• update the Docker image name in the existing runtime config
• create a pipeline and add a notebook
• Configure the notebook properties and select the (updated) runtime config
• Inspect the .pipeline file

expected behavior: the updated value is embedded in the pipeline file observed behavior: the old value is embedded in the pipeline file

[ptitzler]

I forgot to mention that in both scenarios the UI was used to create the runtime image configuration.

[kevin-bates]

I'm sorry, you're right. I misread the opening description thinking the pipeline had previously existed with the original runtime-image referenced (and prior to its update).

Yeah, any updates made from the UI should unconditionally update the respective cache.

[lresende]

Duplicate of #946

[ptitzler]

I don't think this is a duplicate. We are looking at two distinct problem scenarios.

[lresende]

Sorry for making it as duplicated, after a second and third read on the issue I think I understand it now. I have changed the issue title to better represent the issue that is happening on the user interface.