Azure / Azure-Orbital-Analytics-Samples

Sample solution that demonstrates how to deploy and analyze spaceborne data using Azure Synapse Analytics
https://aka.ms/synapse-geospatial-analytics
MIT License
30 stars 24 forks source link

Copy Xml/JSON: The specified resource does not exist #82

Closed ivo-andreev closed 1 year ago

ivo-andreev commented 2 years ago

The synapse pipeline was created by searching in the gallery for "Spaceborne Data Analysis Master Pipeline" and setting the 3 required linked service accordingly:

  1. A linked service to AZ blob storage rawdata**
  2. A linked service to AZ datalake storage v.2 rawdata**
  3. A linked service to

While running the Custom Vision Pipeline there is the following error (similar for Copy XML and Copy JSON activities): It is unclear which the resource is so that

{ "errorCode": "2200", "message": "ErrorCode=UserErrorFileNotFound,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error Message: The specified resource does not exist. (ErrorCode: 404, Detail: The specified resource does not exist., RequestId: 039b4525-001a-0061-2f79-d3625a000000),Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Azure.Storage.StorageException,Message=The specified resource does not exist.,Source=Microsoft.Azure.Storage.Common,'", "failureType": "UserError", "target": "Copy Xml", "details": [] }

The activity is using "@concat(pipeline().parameters.Prefix, '/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['resultsDirectory'], '/other'

The results directory from Read Spec Document activity is "out" and the prefix is the container name, hence the path shall be "-test-container/out/other"

A hint about the configuration is more than welcome as the directory was not there and nothing changed after creating it manually. @jfrazee Do you think you could give a hint?

mandarinamdar commented 2 years ago

Thanks @ivo-andreev for trying out the samples and reporting this. @senthilkungumaraj - can you look into this issue?

ivo-andreev commented 2 years ago

Thanks, this could be my mistake, although it is not trivial and probably some aditional information about what is the file/path would be helpful. I have certain expectations as stated. Looking forward.

karthick-rn commented 2 years ago

@ivo-andreev, The pipeline template in the gallery only supports custom-vision-model-v2 and I think you're trying to run the custom-vision-model (older version) which I doubt will work. Also, going forward we'll not have 2 versions as it seems a bit confusing and we want customers to pivot towards v2. Currently, we are working on removing the instructions for the older version and updating the overall documentation. Ideally, the flow is to deploy the infrastructure, perform the required configuration and generate the v2 package - all this is done via the setup.sh script. Once the package is ready, if you are using the pipeline template from Synapse gallery then all the instructions you need to configure the pipeline is available in this link - https://github.com/Azure/Azure-Orbital-Analytics-Samples/blob/main/deploy/gallery/instructions.md.

There are currently a few PR's being reviewed and we aim to get our documentation updated by end of this week. If you are ok to wait until then great otherwise you can follow the instructions under the gallery folder to configure the v2 related parameters for the pipeline.

Let me know if any questions?

ivo-andreev commented 2 years ago

@karthick-rn I see changes from yesterday evening, just as you said, and will check them immediately I am only worried that after that many attempts my environment would be a mess and am trying to avoid starting from scratch

Thanks for the swift reaction.

karthick-rn commented 2 years ago

@karthick-rn I see changes from yesterday evening, just as you said, and will check them immediately I am only worried that after that many attempts my environment would be a mess and am trying to avoid starting from scratch

Thanks for the swift reaction.

@ivo-andreev, we have a few more changes in flight to complete the documentation work. I will update you here once it is ready. Thanks

karthick-rn commented 2 years ago

@ivo-andreev, We have completed updating our documentation. If you want to try, please give it a go and let us know your feedback. Thanks!