Consolidation_OneEntity Pipeline JSON error

[erik-stefanov]

Hello,

When trying to implement the Consolidation_OneEntity pipeline we get the following error when trying to save it: Could not load resource 'Consolidation_OneEntity'. Please ensure no mistakes in the JSON and that referenced resources exist.

Could you please assist?

[Bertverbeek4PS]

Hi @erik-stefanov importing the json files must be in this order:

Sequence # | Name & URL | Tab | Menu to invoke under the + sign | Help doc -- | -- | -- | -- | -- 1 | data_dataset | Data | Integration dataset | Create an integration dataset 2 | data_dataset_parquet | Data | Integration dataset | Create an integration dataset 3 | dataManifest_dataset | Data | Integration dataset | Create an integration dataset 4 | deltas_dataset | Data | Integration dataset | Create an integration dataset 5 | deltasManifest_dataset | Data | Integration dataset | Create an integration dataset 6 | entity_dataset | Data | Integration dataset | Create an integration dataset 7 | staging_dataset | Data | Integration dataset | Create an integration dataset 8 | Consolidation_flow | Develop | Data flow | Create a data flow 9 | Consolidation_OneEntity | Integrate | Pipeline | Create a pipeline 10 | Consolidation_CheckForDeltas | Integrate | Pipeline | Create a pipeline 11 | Consolidation_AllEntities | Integrate | Pipeline | Create a pipeline Sequence # Name & URL Tab Menu to invoke under the + sign Help doc 1 [data_dataset](https://github.com/Bertverbeek4PS/bc2adls/blob/main/synapse/dataset/data_dataset.json) Data Integration dataset [Create an integration dataset](https://learn.microsoft.com/en-us/azure/data-factory/concepts-datasets-linked-services?tabs=synapse-analytics#tabpanel_1_synapse-analytics) 2 [data_dataset_parquet](https://github.com/Bertverbeek4PS/bc2adls/blob/main/synapse/dataset/data_dataset_parquet.json) Data Integration dataset [Create an integration dataset](https://learn.microsoft.com/en-us/azure/data-factory/concepts-datasets-linked-services?tabs=synapse-analytics#tabpanel_1_synapse-analytics) 3 [dataManifest_dataset](https://github.com/Bertverbeek4PS/bc2adls/blob/main/synapse/dataset/dataManifest_dataset.json) Data Integration dataset [Create an integration dataset](https://learn.microsoft.com/en-us/azure/data-factory/concepts-datasets-linked-services?tabs=synapse-analytics#tabpanel_1_synapse-analytics) 4 [deltas_dataset](https://github.com/Bertverbeek4PS/bc2adls/blob/main/synapse/dataset/deltas_dataset.json) Data Integration dataset [Create an integration dataset](https://learn.microsoft.com/en-us/azure/data-factory/concepts-datasets-linked-services?tabs=synapse-analytics#tabpanel_1_synapse-analytics) 5 [deltasManifest_dataset](https://github.com/Bertverbeek4PS/bc2adls/blob/main/synapse/dataset/deltasManifest_dataset.json) Data Integration dataset [Create an integration dataset](https://learn.microsoft.com/en-us/azure/data-factory/concepts-datasets-linked-services?tabs=synapse-analytics#tabpanel_1_synapse-analytics) 6 [entity_dataset](https://github.com/Bertverbeek4PS/bc2adls/blob/main/synapse/dataset/entity_dataset.json) Data Integration dataset [Create an integration dataset](https://learn.microsoft.com/en-us/azure/data-factory/concepts-datasets-linked-services?tabs=synapse-analytics#tabpanel_1_synapse-analytics) 7 [staging_dataset](https://github.com/Bertverbeek4PS/bc2adls/blob/main/synapse/dataset/staging_dataset.json) Data Integration dataset [Create an integration dataset](https://learn.microsoft.com/en-us/azure/data-factory/concepts-datasets-linked-services?tabs=synapse-analytics#tabpanel_1_synapse-analytics) 8 [Consolidation_flow](https://github.com/Bertverbeek4PS/bc2adls/blob/main/synapse/dataflow/Consolidation_flow.json) Develop Data flow [Create a data flow](https://learn.microsoft.com/en-us/azure/synapse-analytics/concepts-data-flow-overview#getting-started) 9 [Consolidation_OneEntity](https://github.com/Bertverbeek4PS/bc2adls/blob/main/synapse/pipeline/Consolidation_OneEntity.json) Integrate Pipeline [Create a pipeline](https://learn.microsoft.com/en-us/azure/data-factory/tutorial-copy-data-portal#create-a-pipeline) 10 [Consolidation_CheckForDeltas](https://github.com/Bertverbeek4PS/bc2adls/blob/main/synapse/pipeline/Consolidation_CheckForDeltas.json) Integrate Pipeline [Create a pipeline](https://learn.microsoft.com/en-us/azure/data-factory/tutorial-copy-data-portal#create-a-pipeline) 11 [Consolidation_AllEntities](https://github.com/Bertverbeek4PS/bc2adls/blob/main/synapse/pipeline/Consolidation_AllEntities.json) Integrate Pipeline [Create a pipeline](https://learn.microsoft.com/en-us/azure/data-factory/tutorial-copy-data-portal#create-a-pipeline)

[erik-stefanov]

Hello,

Yes, I am aware of the sequence. Here are all the previous steps created:

However the Consolidation_OneEntity pipeline cannot be created because of the above mentioned issue.

[Bertverbeek4PS]

Do you have published the previous datasets? (it is a while ago that I have installed the synapse pipelines). Otherwise maybe setup a meeting?

[erik-stefanov]

Yes, everything has been published. I think a quick meeting would be helpful. Let me know a suitable time frame for you.

[Bertverbeek4PS]

Hi @erik-stefanov i can do tomorrow afternoon 13:00-17:00 (UTC+1)

[erik-stefanov]

Hi, lets meet at 13:00 https://teams.microsoft.com/l/meetup-join/19%3ameeting_YjdiOTk0N2QtMDhjZS00NzM5LWFjN2YtYjIzNGI5Zjk0Yjlj%40thread.v2/0?context=%7b%22Tid%22%3a%2242151053-0193-47aa-9e81-effd81f772cc%22%2c%22Oid%22%3a%2296eaf611-fe6b-4b9d-bfab-dc94e248d91f%22%7d

[Bertverbeek4PS]

Creation of notebook wasn;t in the order in the setup manual. Fixed that with pull request: https://github.com/Bertverbeek4PS/bc2adls/pull/58