search

microsoft / azure-data-manager-for-energy-experience-lab

Experience Lab is a set of utilities that assist in creating instances of Microsoft Azure Data Manager for Energy, performing data loads, and performing basic management operations.
MIT License
13 stars 7 forks source link

while i am running the template spec to upload the data to ADME instance getting the access denied error. #37

Closed sreedharguda closed 1 year ago

sreedharguda commented 1 year ago

While i am running the template spec to upload the data to ADME instance getting the access denied error. Please help me how to mitigate this issue.

2023-07-05 06:06:20,558 [Dataload ] [INFO ] args: Namespace(dir='/app/open-test-data/datasets/documents', output='output/loaded-documents-datasets.json', subparser='datasets') 2023-07-05 06:06:20,614 [Dataload ] [INFO ] Amount of data to transfer: 26.22MB 2023-07-05 06:06:20,614 [Dataload ] [INFO ] Request worker count: 50 2023-07-05 06:06:20,614 [Dataload ] [INFO ] Max Chunk size: 32 MB 2023-07-05 06:06:20,654 [Dataload ] [INFO ] Total number of files to upload: 9 2023-07-05 06:06:20,870 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/FP.pdf with response <Response [403]> 2023-07-05 06:06:20,884 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/osduka_demo_texts.txt with response <Response [403]> 2023-07-05 06:06:20,977 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/prov22.pdf with response <Response [403]> 2023-07-05 06:06:21,108 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/FS12-3031.pdf with response <Response [403]> 2023-07-05 06:06:21,119 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/FI.pdf with response <Response [403]> 2023-07-05 06:06:21,121 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/69_D_CH_11.pdf with response <Response [403]> 2023-07-05 06:06:21,124 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/prov33.pdf with response <Response [403]> 2023-07-05 06:06:21,149 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/FS12-3003.pdf with response <Response [403]> 2023-07-05 06:06:21,539 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/69_D_CH_13.pdf with response <Response [403]> 2023-07-05 06:06:21,674 [Dataload ] [INFO ] File location map is saved to output/loaded-documents-datasets.json 2023-07-05 06:06:21,690 [Dataload ] [INFO ] Files that could not be uploaded: 9 2023-07-05 06:06:21,690 [Dataload ] [INFO ] ['/app/open-test-data/datasets/documents/FP.pdf', '/app/open-test-data/datasets/documents/osduka_demo_texts.txt', '/app/open-test-data/datasets/documents/prov22.pdf', '/app/open-test-data/datasets/documents/FS12-3031.pdf', '/app/open-test-data/datasets/documents/FI.pdf',

Originally posted by @sreedharguda in https://github.com/microsoft/azure-data-manager-for-energy-experience-lab/issues/34#issuecomment-1621087801

Miller-Kyle commented 1 year ago

This is likely due to the recent data partition name changes in ADME. I'll need to update Experience Lab to not prepend the instance name to the data partition id.

sreedharguda commented 1 year ago

Hi @Miller-Kyle , Have used your feature branch and created the resources by using bicep template, its created experince lab resource group and ran experience-lab Container instance, its created the "experience-lab" resource group. I ran the Open-test-Data template spec its asked only "Instance Identifier", "Loglevel", didnt asked other details the way it ask when we use TNO data load repo template spec. I am suspecting this could be the reason to get access denied error.

2023-07-07 03:30:53,060 [Dataload ] [INFO ] Max Chunk size: 32 MB 2023-07-07 03:30:53,088 [Dataload ] [INFO ] Total number of files to upload: 9 2023-07-07 03:30:55,422 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/FS12-3003.pdf with response <Response [403]> 2023-07-07 03:30:55,435 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/FI.pdf with response <Response [403]> 2023-07-07 03:30:55,436 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/prov22.pdf with response <Response [403]> 2023-07-07 03:30:55,437 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/69_D_CH_13.pdf with response <Response [403]> 2023-07-07 03:30:55,548 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/69_D_CH_11.pdf with response <Response [403]> 2023-07-07 03:30:55,550 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/FS12-3031.pdf with response <Response [403]> 2023-07-07 03:30:55,550 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/FP.pdf with response <Response [403]> 2023-07-07 03:30:55,558 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/osduka_demo_texts.txt with response <Response [403]> 2023-07-07 03:30:55,559 [Dataload ] [ERROR ] /files/uploadURL failed for /app/open-test-data/datasets/documents/prov33.pdf with response <Response [403]> 2023-07-07 03:30:55,702 [Dataload ] [INFO ] File location map is saved to output/loaded-documents-datasets.json 2023-07-07 03:30:55,713 [Dataload ] [INFO ] F

sreedharguda commented 1 year ago

If i use this Repo https://github.com/Azure/osdu-data-load-tno and use for data load template spec file it ask all the details of ADME instance name, Client ID, partition name... once we run the container instance the data load works on ADME instance.

Miller-Kyle commented 1 year ago

Fixed in release v0.9

sreedharguda commented 1 year ago

Thank you for your help @Miller-Kyle , have tested and its working fine.