search

Azure / apiops

APIOps applies the concepts of GitOps and DevOps to API deployment. By using practices from these two methodologies, APIOps can enable everyone involved in the lifecycle of API design, development, and deployment with self-service and automated tools to ensure the quality of the specifications and APIs that they’re building.
https://azure.github.io/apiops
MIT License
304 stars 179 forks source link

Getting No such file or directory error while running extractor #113

Closed HakarK closed 1 year ago

HakarK commented 1 year ago

This error occur while the script is trying to make a pull request. image

hamedy commented 1 year ago

I'm also getting the same error

hamedy commented 1 year ago

seems like nothing is downloaded in the previous step

Download artifact to: /home/vsts/work/1/artifacts-from-portal Using default max parallelism. Max dedup parallelism: 192 Verbose, ArtifactHttpClientFactory.CreateVssHttpClient: DedupStoreHttpClient with BaseUri: https://vsblobprodsu6weu.vsblob.visualstudio.com/A398fa71b-6529-4be3-a7e2-cb500435ad5e/, MaxRetries:5, SendTimeout:00:05:00 Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 8a226341-97fe-46b9-8df4-ff52152e2532 Verbose, Started ITelemetrySender BlobStoreApplicationInsightsTelemetrySender. Verbose, PipelineArtifact.DownloadMultipleArtifactsAsync starting Information, DedupManifestArtifactClient will correlate http requests with X-TFS-Session 8a226341-97fe-46b9-8df4-ff52152e2532 Information, Minimatch patterns: [**] Information, Filtered 0 files from the Minimatch filters supplied. Verbose, Could not initialize dataport: Dataport will not be used as it is only currently available on Windows. Information, Downloaded 0.0 MB out of 0.0 MB (100%). Information, Downloaded 0.0 MB out of 0.0 MB (100%). Information, Download statistics: Total Content: 0.0 MB Physical Content Downloaded: 0.0 MB Compression Saved: 0.0 MB Local Caching Saved: 0.0 MB Chunks Downloaded: 0 Nodes Downloaded: 0

waelkdouh commented 1 year ago

I would check ADO to see if any artifacts were generated by the extractor.

If artifacts were generated, then it's a pipeline issue. If no artifacts were generated, then it points towards the extractor.

hamedy commented 1 year ago

under artifacts there's only the extractor, if that's where I should look? image

hamedy commented 1 year ago

the extractor runs successfully with no errors but the publish pipeline artifact step (which also runs successfully with no errors) doesn't seem to publish the content. total content size is less than 1kb while a similar APIM instance in a working environment seems to be around 25KB.

Information, Content upload is done! Information, Content upload statistics: Total Content: 61.0 bytes Physical Content Uploaded: 0.0 bytes Logical Content Uploaded: 0.0 bytes Compression Saved: 0.0 bytes Deduplication Saved: 61.0 bytes Number of Chunks Uploaded: 0 Total Number of Chunks: 1

waelkdouh commented 1 year ago

image

image

hamedy commented 1 year ago

I think I caused some confusion. I'm not trying to publish to APIM. the issue is with extracting APIM using the extractor. the "Publish pipeline artifact" step seems to not publish content to the pipeline directory. So pull request step fails because there's no files inside the output path.

image

hamedy commented 1 year ago

with a out of the box instance of APIM the content upload should be like 20+ KB but in our case it's just 61 bytes. But in the "Run extractor" step I can see the tool reaching our APIM resource and the API's.

hamedy commented 1 year ago

please read my post above one more time, our issue is that the pull request fails because there's no artifacts to make a PR for. Therefore there's nothing in the repo.

waelkdouh commented 1 year ago

I read it multiple times. I am checking with the lead developer and will get back to you. Please be patient.

guythetechie commented 1 year ago

Hi @hamedy,

The Run extractor step should have generated an artifact. It sounds like nothing was generated. Can you click on the pipeline job and the generated artifact? image

Expanding that should look like this: image

Please confirm if that artifact has anything in it.

colincmac commented 1 year ago

If possible, @hamedy & @HakarK , can you also share screen shots of the Run extractor logs and share your pipeline YAML files (minus any sensitive information).

hamedy commented 1 year ago

the artifact is artifacts-from-portal.zip and inside the zip there's nothing. I'm using the Yaml files from API-Ops without any changes. I sat up API-Ops in a test tenant and everything worked great. Where we have the issues is at a customer tenant and we have the APIM inside a vnet, if that makes a difference? From the logs seems like there's no issues reaching the APIM instance because in the logs I can see our API's from APIM. I'll add the run extractor logs below.

hamedy commented 1 year ago

2022-10-21T12:46:02.6172376Z ##[debug]Evaluating condition for step: 'Run extractor' 2022-10-21T12:46:02.6173904Z ##[debug]Evaluating: SucceededNode() 2022-10-21T12:46:02.6174463Z ##[debug]Evaluating SucceededNode: 2022-10-21T12:46:02.6175448Z ##[debug]=> True 2022-10-21T12:46:02.6176042Z ##[debug]Result: True 2022-10-21T12:46:02.6176521Z ##[section]Starting: Run extractor 2022-10-21T12:46:02.6186833Z ============================================================================== 2022-10-21T12:46:02.6187194Z Task : Azure CLI 2022-10-21T12:46:02.6187682Z Description : Run Azure CLI commands against an Azure subscription in a PowerShell Core/Shell script when running on Linux agent or PowerShell/PowerShell Core/Batch script when running on Windows agent. 2022-10-21T12:46:02.6188227Z Version : 2.208.0 2022-10-21T12:46:02.6188437Z Author : Microsoft Corporation 2022-10-21T12:46:02.6188993Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-cli 2022-10-21T12:46:02.6189366Z ============================================================================== 2022-10-21T12:46:02.6332715Z ##[debug]Using node path: /home/vsts/agents/2.211.1/externals/node10/bin/node 2022-10-21T12:46:02.7370258Z ##[debug]agent.TempDirectory=/home/vsts/work/_temp 2022-10-21T12:46:02.7393234Z ##[debug]loading inputs and endpoints 2022-10-21T12:46:02.7442145Z ##[debug]loading INPUT_CONNECTEDSERVICENAMEARM 2022-10-21T12:46:02.7443576Z ##[debug]loading INPUT_SCRIPTTYPE 2022-10-21T12:46:02.7444324Z ##[debug]loading INPUT_SCRIPTLOCATION 2022-10-21T12:46:02.7445020Z ##[debug]loading INPUT_SCRIPTPATH 2022-10-21T12:46:02.7445711Z ##[debug]loading INPUT_INLINESCRIPT 2022-10-21T12:46:02.7446412Z ##[debug]loading INPUT_POWERSHELLERRORACTIONPREFERENCE 2022-10-21T12:46:02.7447153Z ##[debug]loading INPUT_ADDSPNTOENVIRONMENT 2022-10-21T12:46:02.7447809Z ##[debug]loading INPUT_USEGLOBALCONFIG 2022-10-21T12:46:02.7448458Z ##[debug]loading INPUT_CWD 2022-10-21T12:46:02.7449135Z ##[debug]loading INPUT_FAILONSTANDARDERROR 2022-10-21T12:46:02.7449870Z ##[debug]loading INPUT_POWERSHELLIGNORELASTEXITCODE 2022-10-21T12:46:02.7451235Z ##[debug]loading ENDPOINT_AUTH_a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea 2022-10-21T12:46:02.7452185Z ##[debug]loading ENDPOINT_AUTH_SCHEME_a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea 2022-10-21T12:46:02.7453115Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea_SERVICEPRINCIPALID 2022-10-21T12:46:02.7454127Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea_AUTHENTICATIONTYPE 2022-10-21T12:46:02.7455061Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea_TENANTID 2022-10-21T12:46:02.7456047Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea_SERVICEPRINCIPALKEY 2022-10-21T12:46:02.7456881Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION 2022-10-21T12:46:02.7457588Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION 2022-10-21T12:46:02.7458341Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN 2022-10-21T12:46:02.7459117Z ##[debug]loading SECRET_AZURE_SUBSCRIPTION_ID 2022-10-21T12:46:02.7459784Z ##[debug]loading SECRET_AZURE_CLIENT_ID 2022-10-21T12:46:02.7460464Z ##[debug]loading SECRET_AZURE_BEARER_TOKEN 2022-10-21T12:46:02.7514772Z ##[debug]loading SECRET_AZURE_TENANT_ID 2022-10-21T12:46:02.7517972Z ##[debug]loading SECRET_SYSTEM_ACCESSTOKEN 2022-10-21T12:46:02.7519448Z ##[debug]loading SECRET_AZURE_CLIENT_SECRET 2022-10-21T12:46:02.7520868Z ##[debug]loaded 26 2022-10-21T12:46:02.7522296Z ##[debug]Agent.ProxyUrl=undefined 2022-10-21T12:46:02.7522888Z ##[debug]Agent.CAInfo=undefined 2022-10-21T12:46:02.7524298Z ##[debug]Agent.ClientCert=undefined 2022-10-21T12:46:02.7524911Z ##[debug]Agent.SkipCertValidation=undefined 2022-10-21T12:46:02.7526050Z ##[debug]check path : /home/vsts/work/_tasks/AzureCLI_46e4be58-730b-4389-8a2f-ea10b3e5e815/2.208.0/task.json 2022-10-21T12:46:02.7527059Z ##[debug]adding resource file: /home/vsts/work/_tasks/AzureCLI_46e4be58-730b-4389-8a2f-ea10b3e5e815/2.208.0/task.json 2022-10-21T12:46:02.7528110Z ##[debug]system.culture=en-US 2022-10-21T12:46:02.7528740Z ##[debug]which 'az' 2022-10-21T12:46:02.7529377Z ##[debug]found: '/usr/bin/az' 2022-10-21T12:46:02.7529937Z ##[debug]scriptType=bash 2022-10-21T12:46:02.7530510Z ##[debug]scriptLocation=inlineScript 2022-10-21T12:46:02.7531412Z ##[debug]scriptArguments=undefined 2022-10-21T12:46:02.7532032Z ##[debug]Agent.TempDirectory=/home/vsts/work/_temp 2022-10-21T12:46:02.7532898Z ##[debug]inlineScript=chmod u+x /home/vsts/work/1/extractor/extractor /home/vsts/work/1/extractor/extractor result=$? echo "Exit code is $result" exit $result 2022-10-21T12:46:02.7533781Z ##[debug]which 'bash' 2022-10-21T12:46:02.7534415Z ##[debug]found: '/usr/bin/bash' 2022-10-21T12:46:02.7535067Z ##[debug]which '/usr/bin/bash' 2022-10-21T12:46:02.7535704Z ##[debug]found: '/usr/bin/bash' 2022-10-21T12:46:02.7536375Z ##[debug]/usr/bin/bash arg: /home/vsts/work/_temp/azureclitaskscript1666356362749.sh 2022-10-21T12:46:02.7537215Z ##[debug]cwd=/home/vsts/work/1/s 2022-10-21T12:46:02.7537806Z ##[debug]scriptLocation=inlineScript 2022-10-21T12:46:02.7538425Z ##[debug]failOnStandardError=true 2022-10-21T12:46:02.7539138Z ##[debug]testing directory '/home/vsts/work/1/s' 2022-10-21T12:46:02.7560126Z ##[debug]which 'az' 2022-10-21T12:46:02.7562145Z ##[debug]found: '/usr/bin/az' 2022-10-21T12:46:02.7562838Z ##[debug]/usr/bin/az arg: --version 2022-10-21T12:46:02.7565732Z ##[debug]/usr/bin/az arg: --version 2022-10-21T12:46:02.7566333Z ##[debug]exec tool: /usr/bin/az 2022-10-21T12:46:02.7566909Z ##[debug]exec tool: /usr/bin/az 2022-10-21T12:46:02.7567457Z ##[debug]arguments: 2022-10-21T12:46:02.7568822Z ##[debug]arguments: 2022-10-21T12:46:02.7569454Z ##[debug] --version 2022-10-21T12:46:02.7570059Z ##[debug] --version 2022-10-21T12:46:02.7570706Z [command]/usr/bin/az --version 2022-10-21T12:46:03.2478854Z azure-cli 2.41.0 2022-10-21T12:46:03.2479663Z 2022-10-21T12:46:03.2480226Z core 2.41.0 2022-10-21T12:46:03.2480783Z telemetry 1.0.8 2022-10-21T12:46:03.2481133Z 2022-10-21T12:46:03.2481611Z Extensions: 2022-10-21T12:46:03.2482336Z azure-devops 0.25.0 2022-10-21T12:46:03.2482728Z 2022-10-21T12:46:03.2483259Z Dependencies: 2022-10-21T12:46:03.2483775Z msal 1.20.0b1 2022-10-21T12:46:03.2484510Z azure-mgmt-resource 21.1.0b1 2022-10-21T12:46:03.2484902Z 2022-10-21T12:46:03.2485589Z Python location '/opt/az/bin/python3' 2022-10-21T12:46:03.2486401Z Extensions directory '/opt/az/azcliextensions' 2022-10-21T12:46:03.2486812Z 2022-10-21T12:46:03.2487421Z Python (Linux) 3.10.5 (main, Oct 10 2022, 03:02:09) [GCC 9.4.0] 2022-10-21T12:46:03.2487888Z 2022-10-21T12:46:03.2488418Z Legal docs and information: aka.ms/AzureCliLegal 2022-10-21T12:46:03.2488823Z 2022-10-21T12:46:03.2489113Z 2022-10-21T12:46:03.2489748Z Your CLI is up-to-date. 2022-10-21T12:46:03.2501875Z ##[debug]useGlobalConfig=false 2022-10-21T12:46:03.2502955Z ##[debug]Agent.TempDirectory=/home/vsts/work/_temp 2022-10-21T12:46:03.2503861Z ##[debug]Agent.TempDirectory=/home/vsts/work/_temp 2022-10-21T12:46:03.2516185Z Setting AZURE_CONFIG_DIR env variable to: /home/vsts/work/_temp/.azclitask 2022-10-21T12:46:03.2521501Z ##[debug]connectedServiceNameARM=a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea 2022-10-21T12:46:03.2523428Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea data environment = AzureCloud 2022-10-21T12:46:03.2525173Z Setting active cloud to: AzureCloud 2022-10-21T12:46:03.2526763Z ##[debug]which 'az' 2022-10-21T12:46:03.2530660Z ##[debug]found: '/usr/bin/az' 2022-10-21T12:46:03.2532022Z ##[debug]/usr/bin/az arg: cloud set -n AzureCloud 2022-10-21T12:46:03.2533120Z ##[debug]/usr/bin/az arg: cloud set -n AzureCloud 2022-10-21T12:46:03.2534366Z ##[debug]exec tool: /usr/bin/az 2022-10-21T12:46:03.2535462Z ##[debug]exec tool: /usr/bin/az 2022-10-21T12:46:03.2536522Z ##[debug]arguments: 2022-10-21T12:46:03.2537563Z ##[debug]arguments: 2022-10-21T12:46:03.2538645Z ##[debug] cloud 2022-10-21T12:46:03.2539723Z ##[debug] cloud 2022-10-21T12:46:03.2540754Z ##[debug] set 2022-10-21T12:46:03.2541789Z ##[debug] set 2022-10-21T12:46:03.2542945Z ##[debug] -n 2022-10-21T12:46:03.2543967Z ##[debug] -n 2022-10-21T12:46:03.2544960Z ##[debug] AzureCloud 2022-10-21T12:46:03.2545991Z ##[debug] AzureCloud 2022-10-21T12:46:03.2548351Z [command]/usr/bin/az cloud set -n AzureCloud 2022-10-21T12:46:03.4642613Z ##[debug]connectedServiceNameARM=a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea 2022-10-21T12:46:03.4645317Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea auth scheme = ServicePrincipal 2022-10-21T12:46:03.4647289Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea data SubscriptionID = 2022-10-21T12:46:03.4648683Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea auth param authenticationType = spnKey 2022-10-21T12:46:03.4650052Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea auth param serviceprincipalid = 2022-10-21T12:46:03.4654652Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea auth param tenantid = 2022-10-21T12:46:03.4655426Z ##[debug]key based endpoint 2022-10-21T12:46:03.4656794Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea auth param serviceprincipalkey = 2022-10-21T12:46:03.4658501Z ##[debug]Processed: ##vso[task.setsecret] 2022-10-21T12:46:03.4659320Z ##[debug]which 'az' 2022-10-21T12:46:03.4661308Z ##[debug]found: '/usr/bin/az' 2022-10-21T12:46:03.4662954Z ##[debug]/usr/bin/az arg: login --service-principal -u "" --password= --tenant "" --allow-no-subscriptions 2022-10-21T12:46:03.4664744Z ##[debug]/usr/bin/az arg: login --service-principal -u "" --password= --tenant "" --allow-no-subscriptions 2022-10-21T12:46:03.4665847Z ##[debug]exec tool: /usr/bin/az 2022-10-21T12:46:03.4667397Z ##[debug]exec tool: /usr/bin/az 2022-10-21T12:46:03.4668239Z ##[debug]arguments: 2022-10-21T12:46:03.4669025Z ##[debug]arguments: 2022-10-21T12:46:03.4669790Z ##[debug] login 2022-10-21T12:46:03.4670531Z ##[debug] login 2022-10-21T12:46:03.4671483Z ##[debug] --service-principal 2022-10-21T12:46:03.4672413Z ##[debug] --service-principal 2022-10-21T12:46:03.4673286Z ##[debug] -u 2022-10-21T12:46:03.4674148Z ##[debug] -u 2022-10-21T12:46:03.4675084Z ##[debug] 2022-10-21T12:46:03.4676035Z ##[debug] 2022-10-21T12:46:03.4677120Z ##[debug] --password= 2022-10-21T12:46:03.4678236Z ##[debug] --password= 2022-10-21T12:46:03.4679138Z ##[debug] --tenant 2022-10-21T12:46:03.4680000Z ##[debug] --tenant 2022-10-21T12:46:03.4680964Z ##[debug] 2022-10-21T12:46:03.4681899Z ##[debug] 2022-10-21T12:46:03.4682807Z ##[debug] --allow-no-subscriptions 2022-10-21T12:46:03.4683740Z ##[debug] --allow-no-subscriptions 2022-10-21T12:46:03.4685222Z [command]/usr/bin/az login --service-principal -u --password= --tenant --allow-no-subscriptions 2022-10-21T12:46:04.1058141Z [ 2022-10-21T12:46:04.1058418Z { 2022-10-21T12:46:04.1058699Z "cloudName": "AzureCloud", 2022-10-21T12:46:04.1059765Z "homeTenantId": "", 2022-10-21T12:46:04.1060269Z "id": "", 2022-10-21T12:46:04.1060543Z "isDefault": true, 2022-10-21T12:46:04.1060818Z "managedByTenants": [], 2022-10-21T12:46:04.1061229Z "name": "Hemit-TPHP-Dev/Test", 2022-10-21T12:46:04.1061517Z "state": "Enabled", 2022-10-21T12:46:04.1062014Z "tenantId": "", 2022-10-21T12:46:04.1062251Z "user": { 2022-10-21T12:46:04.1062822Z "name": "", 2022-10-21T12:46:04.1063101Z "type": "servicePrincipal" 2022-10-21T12:46:04.1063339Z } 2022-10-21T12:46:04.1063547Z } 2022-10-21T12:46:04.1063733Z ] 2022-10-21T12:46:04.1064441Z ##[debug]which 'az' 2022-10-21T12:46:04.1068694Z ##[debug]found: '/usr/bin/az' 2022-10-21T12:46:04.1069503Z ##[debug]/usr/bin/az arg: account set --subscription "" 2022-10-21T12:46:04.1070348Z ##[debug]/usr/bin/az arg: account set --subscription "" 2022-10-21T12:46:04.1070969Z ##[debug]exec tool: /usr/bin/az 2022-10-21T12:46:04.1071562Z ##[debug]exec tool: /usr/bin/az 2022-10-21T12:46:04.1072110Z ##[debug]arguments: 2022-10-21T12:46:04.1072911Z ##[debug]arguments: 2022-10-21T12:46:04.1073454Z ##[debug] account 2022-10-21T12:46:04.1073978Z ##[debug] account 2022-10-21T12:46:04.1074492Z ##[debug] set 2022-10-21T12:46:04.1075001Z ##[debug] set 2022-10-21T12:46:04.1075639Z ##[debug] --subscription 2022-10-21T12:46:04.1076262Z ##[debug] --subscription 2022-10-21T12:46:04.1076961Z ##[debug] 2022-10-21T12:46:04.1077625Z ##[debug] 2022-10-21T12:46:04.1078290Z [command]/usr/bin/az account set --subscription *** 2022-10-21T12:46:04.3295322Z ##[debug]addSpnToEnvironment=true 2022-10-21T12:46:04.3297184Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea auth scheme = ServicePrincipal 2022-10-21T12:46:04.3304769Z ##[debug]exec tool: /usr/bin/bash 2022-10-21T12:46:04.3305650Z ##[debug]arguments: 2022-10-21T12:46:04.3306550Z ##[debug] /home/vsts/work/_temp/azureclitaskscript1666356362749.sh 2022-10-21T12:46:04.3307628Z [command]/usr/bin/bash /home/vsts/work/_temp/azureclitaskscript1666356362749.sh 2022-10-21T12:46:04.7227253Z info: extractor.Extractor[0] 2022-10-21T12:46:04.7228255Z Beginning execution... 2022-10-21T12:46:04.8053224Z info: Microsoft.Hosting.Lifetime[0] 2022-10-21T12:46:04.8054880Z Application started. Press Ctrl+C to shut down. 2022-10-21T12:46:04.8103356Z info: Microsoft.Hosting.Lifetime[0] 2022-10-21T12:46:04.8104148Z Hosting environment: Production 2022-10-21T12:46:04.8104904Z info: Microsoft.Hosting.Lifetime[0] 2022-10-21T12:46:04.8105497Z Content root path: /home/vsts/work/1/s 2022-10-21T12:46:05.2584689Z info: extractor.Extractor[0] 2022-10-21T12:46:05.2585831Z Exporting service policy... 2022-10-21T12:46:05.4211468Z info: extractor.Extractor[0] 2022-10-21T12:46:05.4213107Z Exporting information for named value epicTokenUrl... 2022-10-21T12:46:05.4214541Z info: extractor.Extractor[0] 2022-10-21T12:46:05.4215700Z Exporting information for named value apimAccessTokenUrl... 2022-10-21T12:46:05.4288294Z info: extractor.Extractor[0] 2022-10-21T12:46:05.4290189Z Exporting information for named value jwtFunctionUrl... 2022-10-21T12:46:05.4333816Z info: extractor.Extractor[0] 2022-10-21T12:46:05.4334683Z Exporting information for named value jwttoken... 2022-10-21T12:46:05.4335446Z info: extractor.Extractor[0] 2022-10-21T12:46:05.4336080Z Exporting information for named value 634812659fd6e818241c3ef2... 2022-10-21T12:46:05.4336917Z info: extractor.Extractor[0] 2022-10-21T12:46:05.4338519Z Exporting information for named value p-func-nea-apim-dev-02-key... 2022-10-21T12:46:05.4339160Z info: extractor.Extractor[0] 2022-10-21T12:46:05.4339616Z Exporting information for named value tokenAcquisitionTimeout... 2022-10-21T12:46:05.4340210Z info: extractor.Extractor[0] 2022-10-21T12:46:05.4340601Z Exporting information for named value workstationPrefix... 2022-10-21T12:46:05.7048751Z info: extractor.Extractor[0] 2022-10-21T12:46:05.7049315Z Exporting information for logger azuremonitor... 2022-10-21T12:46:05.7068656Z info: extractor.Extractor[0] 2022-10-21T12:46:05.7069366Z Exporting information for logger p-appinsight-nea-apim-dev... 2022-10-21T12:46:05.8555327Z info: extractor.Extractor[0] 2022-10-21T12:46:05.8555944Z Exporting information for product starter... 2022-10-21T12:46:05.8556550Z info: extractor.Extractor[0] 2022-10-21T12:46:05.8556973Z Exporting information for product lws... 2022-10-21T12:46:06.1424697Z info: extractor.Extractor[0] 2022-10-21T12:46:06.1425596Z Exporting apis for product lws... 2022-10-21T12:46:06.1435244Z info: extractor.Extractor[0] 2022-10-21T12:46:06.1437845Z Exporting information for product unlimited... 2022-10-21T12:46:06.2087948Z info: extractor.Extractor[0] 2022-10-21T12:46:06.2088856Z Exporting policy for product starter... 2022-10-21T12:46:06.5790627Z info: extractor.Extractor[0] 2022-10-21T12:46:06.5791628Z Exporting information for api lws-records... 2022-10-21T12:46:06.5794504Z info: extractor.Extractor[0] 2022-10-21T12:46:06.5795290Z Exporting information for api lws-records;rev=2... 2022-10-21T12:46:06.7196960Z info: extractor.Extractor[0] 2022-10-21T12:46:06.7197932Z Exporting policy for api lws-records... 2022-10-21T12:46:06.7214573Z info: extractor.Extractor[0] 2022-10-21T12:46:06.7215298Z Exporting policy for api lws-records;rev=2... 2022-10-21T12:46:06.7232128Z info: extractor.Extractor[0] 2022-10-21T12:46:06.7233028Z Exporting specification for api lws-records... 2022-10-21T12:46:06.7233806Z info: extractor.Extractor[0] 2022-10-21T12:46:06.7234571Z Exporting specification for api lws-records;rev=2... 2022-10-21T12:46:06.9838136Z info: System.Net.Http.HttpClient.NonAuthenticatedHttpClient.LogicalHandler[100]

***** deleted some stuff here

2022-10-21T12:46:07.1678531Z info: System.Net.Http.HttpClient.NonAuthenticatedHttpClient.ClientHandler[101] 2022-10-21T12:46:07.1679635Z Received HTTP response headers after 167.6115ms - 200 2022-10-21T12:46:07.1680519Z info: System.Net.Http.HttpClient.NonAuthenticatedHttpClient.ClientHandler[101] 2022-10-21T12:46:07.1681289Z Received HTTP response headers after 168.0119ms - 200 2022-10-21T12:46:07.1729387Z info: System.Net.Http.HttpClient.NonAuthenticatedHttpClient.LogicalHandler[101] 2022-10-21T12:46:07.1730690Z End processing HTTP request after 197.1941ms - 200 2022-10-21T12:46:07.1731534Z info: System.Net.Http.HttpClient.NonAuthenticatedHttpClient.LogicalHandler[101] 2022-10-21T12:46:07.1732305Z End processing HTTP request after 197.189ms - 200 2022-10-21T12:46:07.5713979Z info: extractor.Extractor[0] 2022-10-21T12:46:07.5715251Z Exporting policy for apiOperation 63358383b3ffc52c7919d35f in api lws-records... 2022-10-21T12:46:07.6165112Z info: extractor.Extractor[0] 2022-10-21T12:46:07.6167269Z Exporting policy for apiOperation add-workstation in api lws-records;rev=2... 2022-10-21T12:46:07.6793281Z info: extractor.Extractor[0] 2022-10-21T12:46:07.6794386Z Exporting policy for apiOperation workstation-details in api lws-records... 2022-10-21T12:46:07.7321700Z info: extractor.Extractor[0] 2022-10-21T12:46:07.7322580Z Exporting policy for apiOperation workstation-details in api lws-records;rev=2... 2022-10-21T12:46:07.7758312Z info: extractor.Extractor[0] 2022-10-21T12:46:07.7760127Z Exporting policy for apiOperation add-workstation in api lws-records... 2022-10-21T12:46:07.7837770Z info: extractor.Extractor[0] 2022-10-21T12:46:07.7839009Z Exporting policy for apiOperation 63358383b3ffc52c7919d35f in api lws-records;rev=2... 2022-10-21T12:46:08.2538556Z info: extractor.Extractor[0] 2022-10-21T12:46:08.2539511Z Exporting information for diagnostic azuremonitor... 2022-10-21T12:46:08.2540683Z info: extractor.Extractor[0] 2022-10-21T12:46:08.2541298Z Exporting information for diagnostic applicationinsights... 2022-10-21T12:46:08.2608368Z info: extractor.Extractor[0] 2022-10-21T12:46:08.2608795Z Execution complete. 2022-10-21T12:46:08.2617240Z info: Microsoft.Hosting.Lifetime[0] 2022-10-21T12:46:08.2617859Z Application is shutting down... 2022-10-21T12:46:08.2978116Z Exit code is 0 2022-10-21T12:46:08.3013485Z ##[debug]Exit code 0 received from tool '/usr/bin/bash' 2022-10-21T12:46:08.3014844Z ##[debug]STDIO streams have closed for tool '/usr/bin/bash' 2022-10-21T12:46:08.3015825Z ##[debug]task result: Succeeded 2022-10-21T12:46:08.3018369Z ##[debug]Processed: ##vso[task.complete result=Succeeded;]Script exited with return code: 0 2022-10-21T12:46:08.3019576Z ##[debug]which 'az' 2022-10-21T12:46:08.3020600Z ##[debug]found: '/usr/bin/az' 2022-10-21T12:46:08.3021514Z ##[debug]/usr/bin/az arg: account clear 2022-10-21T12:46:08.3022427Z ##[debug]/usr/bin/az arg: account clear 2022-10-21T12:46:08.3023288Z ##[debug]exec tool: /usr/bin/az 2022-10-21T12:46:08.3024168Z ##[debug]exec tool: /usr/bin/az 2022-10-21T12:46:08.3025006Z ##[debug]arguments: 2022-10-21T12:46:08.3025817Z ##[debug]arguments: 2022-10-21T12:46:08.3026639Z ##[debug] account 2022-10-21T12:46:08.3027439Z ##[debug] account 2022-10-21T12:46:08.3028233Z ##[debug] clear 2022-10-21T12:46:08.3029038Z ##[debug] clear 2022-10-21T12:46:08.3029653Z [command]/usr/bin/az account clear 2022-10-21T12:46:08.6025682Z ##[section]Finishing: Run extractor

hamedy commented 1 year ago

@guythetechie @colincmac hey guys, have you managed to think of anything that can help us figure this out?

guythetechie commented 1 year ago

Hi @hamedy

After this line, https://github.com/Azure/apiops/blob/75dadd209a44409982ca78fcfd6590027697c3e1/tools/pipelines/run-extractor.yaml#L78 can you add ls "$(Build.ArtifactStagingDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}".

That path is defined here https://github.com/Azure/apiops/blob/75dadd209a44409982ca78fcfd6590027697c3e1/tools/pipelines/run-extractor.yaml#L92 and should be where the extractor outputs its artifacts.

The idea is to validate whether the extraction was successful.

hamedy commented 1 year ago

It fails with this error: ls: cannot access '/home/vsts/work/1/a/Output': No such file or directory

image

waelkdouh commented 1 year ago

@hamedy I will wait on @guythetechie to respond here but in the meantime let me know if the following suggestion from a previous user would work for you:

"I had a similar issue to this as well when I created the initial repo with fork or import. The fix was to create a new repo and copy the initial files in which resolved it for me. I didn't dig any deeper as that seemed to fix the issue"

guythetechie commented 1 year ago

@hamedy - I've added extra logging to the extractor. Can you do the following:

  1. Replace tools/code/common/CommonTypes.cs with this modified version.
  2. Rerun the Publish extractor pipeline.
  3. Rerun the Run extractor pipeline.
  4. Send the output of the Run extractor step. It should contain additional logging.
hamedy commented 1 year ago

@hamedy - I've added extra logging to the extractor. Can you do the following:

  1. Replace tools/code/common/CommonTypes.cs with this modified version.
  2. Rerun the Publish extractor pipeline.
  3. Rerun the Run extractor pipeline.
  4. Send the output of the Run extractor step. It should contain additional logging.

here's the log:

2022-10-25T19:18:15.1311685Z ##[debug]Evaluating condition for step: 'Run extractor' 2022-10-25T19:18:15.1313701Z ##[debug]Evaluating: SucceededNode() 2022-10-25T19:18:15.1314211Z ##[debug]Evaluating SucceededNode: 2022-10-25T19:18:15.1315257Z ##[debug]=> True 2022-10-25T19:18:15.1315759Z ##[debug]Result: True 2022-10-25T19:18:15.1316226Z ##[section]Starting: Run extractor 2022-10-25T19:18:15.1328167Z ============================================================================== 2022-10-25T19:18:15.1328472Z Task : Azure CLI 2022-10-25T19:18:15.1328982Z Description : Run Azure CLI commands against an Azure subscription in a PowerShell Core/Shell script when running on Linux agent or PowerShell/PowerShell Core/Batch script when running on Windows agent. 2022-10-25T19:18:15.1329477Z Version : 2.208.0 2022-10-25T19:18:15.1329704Z Author : Microsoft Corporation 2022-10-25T19:18:15.1330023Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-cli 2022-10-25T19:18:15.1330606Z ============================================================================== 2022-10-25T19:18:15.1470298Z ##[debug]Using node path: /home/vsts/agents/2.211.1/externals/node10/bin/node 2022-10-25T19:18:15.2614203Z ##[debug]agent.TempDirectory=/home/vsts/work/_temp 2022-10-25T19:18:15.2638728Z ##[debug]loading inputs and endpoints 2022-10-25T19:18:15.2647814Z ##[debug]loading INPUT_CONNECTEDSERVICENAMEARM 2022-10-25T19:18:15.2660048Z ##[debug]loading INPUT_SCRIPTTYPE 2022-10-25T19:18:15.2672387Z ##[debug]loading INPUT_SCRIPTLOCATION 2022-10-25T19:18:15.2675432Z ##[debug]loading INPUT_SCRIPTPATH 2022-10-25T19:18:15.2676041Z ##[debug]loading INPUT_INLINESCRIPT 2022-10-25T19:18:15.2676671Z ##[debug]loading INPUT_POWERSHELLERRORACTIONPREFERENCE 2022-10-25T19:18:15.2677318Z ##[debug]loading INPUT_ADDSPNTOENVIRONMENT 2022-10-25T19:18:15.2678857Z ##[debug]loading INPUT_USEGLOBALCONFIG 2022-10-25T19:18:15.2684376Z ##[debug]loading INPUT_CWD 2022-10-25T19:18:15.2684969Z ##[debug]loading INPUT_FAILONSTANDARDERROR 2022-10-25T19:18:15.2685629Z ##[debug]loading INPUT_POWERSHELLIGNORELASTEXITCODE 2022-10-25T19:18:15.2686831Z ##[debug]loading ENDPOINT_AUTH_a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea 2022-10-25T19:18:15.2698588Z ##[debug]loading ENDPOINT_AUTH_SCHEME_a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea 2022-10-25T19:18:15.2699875Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea_SERVICEPRINCIPALID 2022-10-25T19:18:15.2701758Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea_AUTHENTICATIONTYPE 2022-10-25T19:18:15.2702713Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea_TENANTID 2022-10-25T19:18:15.2704733Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea_SERVICEPRINCIPALKEY 2022-10-25T19:18:15.2705507Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION 2022-10-25T19:18:15.2706179Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION 2022-10-25T19:18:15.2706905Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN 2022-10-25T19:18:15.2707637Z ##[debug]loading SECRET_AZURE_TENANT_ID 2022-10-25T19:18:15.2708247Z ##[debug]loading SECRET_AZURE_CLIENT_ID 2022-10-25T19:18:15.2708868Z ##[debug]loading SECRET_AZURE_SUBSCRIPTION_ID 2022-10-25T19:18:15.2709506Z ##[debug]loading SECRET_SYSTEM_ACCESSTOKEN 2022-10-25T19:18:15.2710125Z ##[debug]loading SECRET_AZURE_BEARER_TOKEN 2022-10-25T19:18:15.2710990Z ##[debug]loading SECRET_AZURE_CLIENT_SECRET 2022-10-25T19:18:15.2711587Z ##[debug]loaded 26 2022-10-25T19:18:15.2712147Z ##[debug]Agent.ProxyUrl=undefined 2022-10-25T19:18:15.2712725Z ##[debug]Agent.CAInfo=undefined 2022-10-25T19:18:15.2713324Z ##[debug]Agent.ClientCert=undefined 2022-10-25T19:18:15.2713932Z ##[debug]Agent.SkipCertValidation=undefined 2022-10-25T19:18:15.2714813Z ##[debug]check path : /home/vsts/work/_tasks/AzureCLI_46e4be58-730b-4389-8a2f-ea10b3e5e815/2.208.0/task.json 2022-10-25T19:18:15.2716849Z ##[debug]adding resource file: /home/vsts/work/_tasks/AzureCLI_46e4be58-730b-4389-8a2f-ea10b3e5e815/2.208.0/task.json 2022-10-25T19:18:15.2717969Z ##[debug]system.culture=en-US 2022-10-25T19:18:15.2718627Z ##[debug]which 'az' 2022-10-25T19:18:15.2724786Z ##[debug]found: '/usr/bin/az' 2022-10-25T19:18:15.2738811Z ##[debug]scriptType=bash 2022-10-25T19:18:15.2739432Z ##[debug]scriptLocation=inlineScript 2022-10-25T19:18:15.2740056Z ##[debug]scriptArguments=undefined 2022-10-25T19:18:15.2742826Z ##[debug]Agent.TempDirectory=/home/vsts/work/_temp 2022-10-25T19:18:15.2743736Z ##[debug]inlineScript=chmod u+x /home/vsts/work/1/extractor/extractor /home/vsts/work/1/extractor/extractor ls /home/vsts/work/1/a/Output result=$? echo "Exit code is $result" exit $result 2022-10-25T19:18:15.2753802Z ##[debug]which 'bash' 2022-10-25T19:18:15.2758157Z ##[debug]found: '/usr/bin/bash' 2022-10-25T19:18:15.2758865Z ##[debug]which '/usr/bin/bash' 2022-10-25T19:18:15.2762013Z ##[debug]found: '/usr/bin/bash' 2022-10-25T19:18:15.2763810Z ##[debug]/usr/bin/bash arg: /home/vsts/work/_temp/azureclitaskscript1666725495274.sh 2022-10-25T19:18:15.2764507Z ##[debug]cwd=/home/vsts/work/1/s 2022-10-25T19:18:15.2765091Z ##[debug]scriptLocation=inlineScript 2022-10-25T19:18:15.2765697Z ##[debug]failOnStandardError=true 2022-10-25T19:18:15.2766435Z ##[debug]testing directory '/home/vsts/work/1/s' 2022-10-25T19:18:15.2799047Z ##[debug]which 'az' 2022-10-25T19:18:15.2801988Z ##[debug]found: '/usr/bin/az' 2022-10-25T19:18:15.2804390Z ##[debug]/usr/bin/az arg: --version 2022-10-25T19:18:15.2805087Z ##[debug]/usr/bin/az arg: --version 2022-10-25T19:18:15.2805703Z ##[debug]exec tool: /usr/bin/az 2022-10-25T19:18:15.2807173Z ##[debug]exec tool: /usr/bin/az 2022-10-25T19:18:15.2807744Z ##[debug]arguments: 2022-10-25T19:18:15.2808289Z ##[debug]arguments: 2022-10-25T19:18:15.2808919Z ##[debug] --version 2022-10-25T19:18:15.2809533Z ##[debug] --version 2022-10-25T19:18:15.2810061Z [command]/usr/bin/az --version 2022-10-25T19:18:15.7856257Z azure-cli 2.41.0 2022-10-25T19:18:15.7857093Z 2022-10-25T19:18:15.7857645Z core 2.41.0 2022-10-25T19:18:15.7858218Z telemetry 1.0.8 2022-10-25T19:18:15.7858624Z 2022-10-25T19:18:15.7859095Z Extensions: 2022-10-25T19:18:15.7859827Z azure-devops 0.25.0 2022-10-25T19:18:15.7860214Z 2022-10-25T19:18:15.7860673Z Dependencies: 2022-10-25T19:18:15.7861191Z msal 1.20.0b1 2022-10-25T19:18:15.7861929Z azure-mgmt-resource 21.1.0b1 2022-10-25T19:18:15.7862311Z 2022-10-25T19:18:15.7862995Z Python location '/opt/az/bin/python3' 2022-10-25T19:18:15.7863809Z Extensions directory '/opt/az/azcliextensions' 2022-10-25T19:18:15.7864213Z 2022-10-25T19:18:15.7864841Z Python (Linux) 3.10.5 (main, Oct 10 2022, 03:02:09) [GCC 9.4.0] 2022-10-25T19:18:15.7865290Z 2022-10-25T19:18:15.7865835Z Legal docs and information: aka.ms/AzureCliLegal 2022-10-25T19:18:15.7866228Z 2022-10-25T19:18:15.7866534Z 2022-10-25T19:18:15.7867175Z Your CLI is up-to-date. 2022-10-25T19:18:15.7878614Z ##[debug]useGlobalConfig=false 2022-10-25T19:18:15.7879889Z ##[debug]Agent.TempDirectory=/home/vsts/work/_temp 2022-10-25T19:18:15.7880951Z ##[debug]Agent.TempDirectory=/home/vsts/work/_temp 2022-10-25T19:18:15.7891114Z Setting AZURE_CONFIG_DIR env variable to: /home/vsts/work/_temp/.azclitask 2022-10-25T19:18:15.7894670Z ##[debug]connectedServiceNameARM=a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea 2022-10-25T19:18:15.7896010Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea data environment = AzureCloud 2022-10-25T19:18:15.7898181Z Setting active cloud to: AzureCloud 2022-10-25T19:18:15.7924421Z ##[debug]which 'az' 2022-10-25T19:18:15.7925464Z ##[debug]found: '/usr/bin/az' 2022-10-25T19:18:15.7927053Z ##[debug]/usr/bin/az arg: cloud set -n AzureCloud 2022-10-25T19:18:15.7927779Z ##[debug]/usr/bin/az arg: cloud set -n AzureCloud 2022-10-25T19:18:15.7928387Z ##[debug]exec tool: /usr/bin/az 2022-10-25T19:18:15.7928968Z ##[debug]exec tool: /usr/bin/az 2022-10-25T19:18:15.7929541Z ##[debug]arguments: 2022-10-25T19:18:15.7930076Z ##[debug]arguments: 2022-10-25T19:18:15.7930902Z ##[debug] cloud 2022-10-25T19:18:15.7931417Z ##[debug] cloud 2022-10-25T19:18:15.7931928Z ##[debug] set 2022-10-25T19:18:15.7932439Z ##[debug] set 2022-10-25T19:18:15.7933057Z ##[debug] -n 2022-10-25T19:18:15.7933643Z ##[debug] -n 2022-10-25T19:18:15.7934171Z ##[debug] AzureCloud 2022-10-25T19:18:15.7934719Z ##[debug] AzureCloud 2022-10-25T19:18:15.7935253Z [command]/usr/bin/az cloud set -n AzureCloud 2022-10-25T19:18:16.0004481Z ##[debug]connectedServiceNameARM=a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea 2022-10-25T19:18:16.0007074Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea auth scheme = ServicePrincipal 2022-10-25T19:18:16.0009097Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea data SubscriptionID = 2022-10-25T19:18:16.0010488Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea auth param authenticationType = spnKey 2022-10-25T19:18:16.0011875Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea auth param serviceprincipalid = 2022-10-25T19:18:16.0013577Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea auth param tenantid = 2022-10-25T19:18:16.0014619Z ##[debug]key based endpoint 2022-10-25T19:18:16.0016346Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea auth param serviceprincipalkey = 2022-10-25T19:18:16.0018468Z ##[debug]Processed: ##vso[task.setsecret] 2022-10-25T19:18:16.0019657Z ##[debug]which 'az' 2022-10-25T19:18:16.0020679Z ##[debug]found: '/usr/bin/az' 2022-10-25T19:18:16.0022228Z ##[debug]/usr/bin/az arg: login --service-principal -u "" --password= --tenant "" --allow-no-subscriptions 2022-10-25T19:18:16.0023895Z ##[debug]/usr/bin/az arg: login --service-principal -u "" --password= --tenant "" --allow-no-subscriptions 2022-10-25T19:18:16.0025058Z ##[debug]exec tool: /usr/bin/az 2022-10-25T19:18:16.0026729Z ##[debug]exec tool: /usr/bin/az 2022-10-25T19:18:16.0027600Z ##[debug]arguments: 2022-10-25T19:18:16.0028393Z ##[debug]arguments: 2022-10-25T19:18:16.0029165Z ##[debug] login 2022-10-25T19:18:16.0029908Z ##[debug] login 2022-10-25T19:18:16.0031138Z ##[debug] --service-principal 2022-10-25T19:18:16.0032112Z ##[debug] --service-principal 2022-10-25T19:18:16.0033014Z ##[debug] -u 2022-10-25T19:18:16.0033859Z ##[debug] -u 2022-10-25T19:18:16.0034818Z ##[debug] 2022-10-25T19:18:16.0035777Z ##[debug] 2022-10-25T19:18:16.0036807Z ##[debug] --password= 2022-10-25T19:18:16.0037858Z ##[debug] --password= 2022-10-25T19:18:16.0038762Z ##[debug] --tenant 2022-10-25T19:18:16.0039647Z ##[debug] --tenant 2022-10-25T19:18:16.0040601Z ##[debug] 2022-10-25T19:18:16.0041542Z ##[debug] 2022-10-25T19:18:16.0042454Z ##[debug] --allow-no-subscriptions 2022-10-25T19:18:16.0043395Z ##[debug] --allow-no-subscriptions 2022-10-25T19:18:16.0044831Z [command]/usr/bin/az login --service-principal -u --password= --tenant --allow-no-subscriptions 2022-10-25T19:18:16.5880401Z [ 2022-10-25T19:18:16.5881085Z { 2022-10-25T19:18:16.5881564Z "cloudName": "AzureCloud", 2022-10-25T19:18:16.5882830Z "homeTenantId": "", 2022-10-25T19:18:16.5883577Z "id": "", 2022-10-25T19:18:16.5883980Z "isDefault": true, 2022-10-25T19:18:16.5884376Z "managedByTenants": [], 2022-10-25T19:18:16.5884925Z "name": "Hemit-TPHP-Dev/Test", 2022-10-25T19:18:16.5885346Z "state": "Enabled", 2022-10-25T19:18:16.5885971Z "tenantId": "", 2022-10-25T19:18:16.5886364Z "user": { 2022-10-25T19:18:16.5887052Z "name": "", 2022-10-25T19:18:16.5887466Z "type": "servicePrincipal" 2022-10-25T19:18:16.5887827Z } 2022-10-25T19:18:16.5888157Z } 2022-10-25T19:18:16.5888483Z ] 2022-10-25T19:18:16.5889396Z ##[debug]which 'az' 2022-10-25T19:18:16.5890221Z ##[debug]found: '/usr/bin/az' 2022-10-25T19:18:16.5891184Z ##[debug]/usr/bin/az arg: account set --subscription "" 2022-10-25T19:18:16.5892195Z ##[debug]/usr/bin/az arg: account set --subscription "" 2022-10-25T19:18:16.5892947Z ##[debug]exec tool: /usr/bin/az 2022-10-25T19:18:16.5893679Z ##[debug]exec tool: /usr/bin/az 2022-10-25T19:18:16.5894380Z ##[debug]arguments: 2022-10-25T19:18:16.5895341Z ##[debug]arguments: 2022-10-25T19:18:16.5896620Z ##[debug] account 2022-10-25T19:18:16.5897230Z ##[debug] account 2022-10-25T19:18:16.5897774Z ##[debug] set 2022-10-25T19:18:16.5898287Z ##[debug] set 2022-10-25T19:18:16.5899241Z ##[debug] --subscription 2022-10-25T19:18:16.5899922Z ##[debug] --subscription 2022-10-25T19:18:16.5900660Z ##[debug] 2022-10-25T19:18:16.5901341Z ##[debug] 2022-10-25T19:18:16.5902041Z [command]/usr/bin/az account set --subscription *** 2022-10-25T19:18:16.8173771Z ##[debug]addSpnToEnvironment=true 2022-10-25T19:18:16.8175678Z ##[debug]a54c2caf-87d7-4ee7-bc4c-33347ecfa8ea auth scheme = ServicePrincipal 2022-10-25T19:18:16.8182974Z ##[debug]exec tool: /usr/bin/bash 2022-10-25T19:18:16.8183853Z ##[debug]arguments: 2022-10-25T19:18:16.8184730Z ##[debug] /home/vsts/work/_temp/azureclitaskscript1666725495274.sh 2022-10-25T19:18:16.8185861Z [command]/usr/bin/bash /home/vsts/work/_temp/azureclitaskscript1666725495274.sh 2022-10-25T19:18:17.2160658Z info: extractor.Extractor[0] 2022-10-25T19:18:17.2161289Z Beginning execution... 2022-10-25T19:18:17.2983326Z info: Microsoft.Hosting.Lifetime[0] 2022-10-25T19:18:17.2984934Z Application started. Press Ctrl+C to shut down. 2022-10-25T19:18:17.3033391Z info: Microsoft.Hosting.Lifetime[0] 2022-10-25T19:18:17.3034204Z Hosting environment: Production 2022-10-25T19:18:17.3035034Z info: Microsoft.Hosting.Lifetime[0] 2022-10-25T19:18:17.3035617Z Content root path: /home/vsts/work/1/s 2022-10-25T19:18:17.8459671Z info: extractor.Extractor[0] 2022-10-25T19:18:17.8464407Z Exporting service policy... 2022-10-25T19:18:17.8471266Z Creating directory /home/vsts/work/1/s/Output... 2022-10-25T19:18:17.8471716Z Overwriting file /home/vsts/work/1/s/Output/policy.xml with text... 2022-10-25T19:18:18.0202705Z info: extractor.Extractor[0] 2022-10-25T19:18:18.0203635Z Exporting information for named value apimAccessTokenUrl... 2022-10-25T19:18:18.0204361Z info: extractor.Extractor[0] 2022-10-25T19:18:18.0204889Z Exporting information for named value epicTokenUrl... 2022-10-25T19:18:18.0259498Z Creating directory /home/vsts/work/1/s/Output/named values/apimAccessTokenUrl... 2022-10-25T19:18:18.0263134Z Creating directory /home/vsts/work/1/s/Output/named values/epicTokenUrl... 2022-10-25T19:18:18.0267031Z Overwriting file /home/vsts/work/1/s/Output/named values/epicTokenUrl/namedValueInformation.json with JSON... 2022-10-25T19:18:18.0268903Z Overwriting file /home/vsts/work/1/s/Output/named values/apimAccessTokenUrl/namedValueInformation.json with JSON... 2022-10-25T19:18:18.0300109Z info: extractor.Extractor[0] 2022-10-25T19:18:18.0300882Z Exporting information for named value jwtFunctionUrl... 2022-10-25T19:18:18.0302142Z Creating directory /home/vsts/work/1/s/Output/named values/jwtFunctionUrl... 2022-10-25T19:18:18.0307164Z Overwriting file /home/vsts/work/1/s/Output/named values/jwtFunctionUrl/namedValueInformation.json with JSON... 2022-10-25T19:18:18.0313534Z info: extractor.Extractor[0] 2022-10-25T19:18:18.0314266Z Exporting information for named value jwttoken... 2022-10-25T19:18:18.0316770Z Creating directory /home/vsts/work/1/s/Output/named values/Logger-Credentials--634812659fd6e818241c3ef3... 2022-10-25T19:18:18.0317705Z info: extractor.Extractor[0] 2022-10-25T19:18:18.0318326Z Exporting information for named value 634812659fd6e818241c3ef2... 2022-10-25T19:18:18.0319031Z Creating directory /home/vsts/work/1/s/Output/named values/jwttoken... 2022-10-25T19:18:18.0323595Z Overwriting file /home/vsts/work/1/s/Output/named values/Logger-Credentials--634812659fd6e818241c3ef3/namedValueInformation.json with JSON... 2022-10-25T19:18:18.0324964Z Overwriting file /home/vsts/work/1/s/Output/named values/jwttoken/namedValueInformation.json with JSON... 2022-10-25T19:18:18.0330822Z info: extractor.Extractor[0] 2022-10-25T19:18:18.0331470Z Exporting information for named value p-func-nea-apim-dev-02-key... 2022-10-25T19:18:18.0332126Z Creating directory /home/vsts/work/1/s/Output/named values/p-func-nea-apim-dev-02-key... 2022-10-25T19:18:18.0333298Z Overwriting file /home/vsts/work/1/s/Output/named values/p-func-nea-apim-dev-02-key/namedValueInformation.json with JSON... 2022-10-25T19:18:18.0339497Z info: extractor.Extractor[0] 2022-10-25T19:18:18.0339949Z Exporting information for named value SOURCEWORKSTATIONNAME... 2022-10-25T19:18:18.0347659Z Creating directory /home/vsts/work/1/s/Output/named values/tokenAcquisitionTimeout... 2022-10-25T19:18:18.0348437Z info: extractor.Extractor[0] 2022-10-25T19:18:18.0349004Z Exporting information for named value tokenAcquisitionTimeout... 2022-10-25T19:18:18.0349529Z Overwriting file /home/vsts/work/1/s/Output/named values/tokenAcquisitionTimeout/namedValueInformation.json with JSON... 2022-10-25T19:18:18.0350083Z Creating directory /home/vsts/work/1/s/Output/named values/SourceWorkstationName... 2022-10-25T19:18:18.0351772Z Overwriting file /home/vsts/work/1/s/Output/named values/SourceWorkstationName/namedValueInformation.json with JSON... 2022-10-25T19:18:18.0355331Z info: extractor.Extractor[0] 2022-10-25T19:18:18.0356574Z Exporting information for named value workstationPrefix... 2022-10-25T19:18:18.0357030Z Creating directory /home/vsts/work/1/s/Output/named values/workstationPrefix... 2022-10-25T19:18:18.0359763Z Overwriting file /home/vsts/work/1/s/Output/named values/workstationPrefix/namedValueInformation.json with JSON... 2022-10-25T19:18:18.2946461Z info: extractor.Extractor[0] 2022-10-25T19:18:18.2947107Z Exporting information for logger azuremonitor... 2022-10-25T19:18:18.2947759Z info: extractor.Extractor[0] 2022-10-25T19:18:18.2948413Z Exporting information for logger p-appinsight-nea-apim-dev... 2022-10-25T19:18:18.2957971Z Creating directory /home/vsts/work/1/s/Output/loggers/azuremonitor... 2022-10-25T19:18:18.2958659Z Overwriting file /home/vsts/work/1/s/Output/loggers/azuremonitor/loggerInformation.json with JSON... 2022-10-25T19:18:18.2968879Z Creating directory /home/vsts/work/1/s/Output/loggers/p-appinsight-nea-apim-dev... 2022-10-25T19:18:18.2970926Z Overwriting file /home/vsts/work/1/s/Output/loggers/p-appinsight-nea-apim-dev/loggerInformation.json with JSON... 2022-10-25T19:18:18.4470773Z info: extractor.Extractor[0] 2022-10-25T19:18:18.4471732Z Exporting information for product starter... 2022-10-25T19:18:18.4473095Z info: extractor.Extractor[0] 2022-10-25T19:18:18.4473579Z Exporting information for product lws... 2022-10-25T19:18:18.4516028Z Creating directory /home/vsts/work/1/s/Output/products/Starter... 2022-10-25T19:18:18.4522905Z Overwriting file /home/vsts/work/1/s/Output/products/Starter/productInformation.json with JSON... 2022-10-25T19:18:18.4531683Z Creating directory /home/vsts/work/1/s/Output/products/lws... 2022-10-25T19:18:18.4535772Z Overwriting file /home/vsts/work/1/s/Output/products/lws/productInformation.json with JSON... 2022-10-25T19:18:18.6168916Z info: extractor.Extractor[0] 2022-10-25T19:18:18.6174244Z Exporting policy for product starter... 2022-10-25T19:18:18.6175574Z Creating directory /home/vsts/work/1/s/Output/products/Starter... 2022-10-25T19:18:18.6176255Z Overwriting file /home/vsts/work/1/s/Output/products/Starter/policy.xml with text... 2022-10-25T19:18:18.7363111Z Creating directory /home/vsts/work/1/s/Output/products/Unlimited... 2022-10-25T19:18:18.7365284Z Overwriting file /home/vsts/work/1/s/Output/products/Unlimited/productInformation.json with JSON... 2022-10-25T19:18:18.7368985Z info: extractor.Extractor[0] 2022-10-25T19:18:18.7371002Z Exporting information for product unlimited... 2022-10-25T19:18:19.6043079Z info: extractor.Extractor[0] 2022-10-25T19:18:19.6044728Z Exporting information for diagnostic azuremonitor... 2022-10-25T19:18:19.6049938Z info: extractor.Extractor[0] 2022-10-25T19:18:19.6051118Z Exporting information for diagnostic applicationinsights... 2022-10-25T19:18:19.6107890Z Creating directory /home/vsts/work/1/s/Output/diagnostics/applicationinsights... 2022-10-25T19:18:19.6112307Z Creating directory /home/vsts/work/1/s/Output/diagnostics/azuremonitor... 2022-10-25T19:18:19.6114339Z Overwriting file /home/vsts/work/1/s/Output/diagnostics/azuremonitor/diagnosticInformation.json with JSON... 2022-10-25T19:18:19.6118482Z Overwriting file /home/vsts/work/1/s/Output/diagnostics/applicationinsights/diagnosticInformation.json with JSON... 2022-10-25T19:18:19.6124932Z info: extractor.Extractor[0] 2022-10-25T19:18:19.6125351Z Execution complete. 2022-10-25T19:18:19.6131560Z info: Microsoft.Hosting.Lifetime[0] 2022-10-25T19:18:19.6132205Z Application is shutting down... 2022-10-25T19:18:19.6493920Z ls: cannot access '/home/vsts/work/1/a/Output': No such file or directory 2022-10-25T19:18:19.6497927Z Exit code is 2 2022-10-25T19:18:19.6502938Z ##[debug]Exit code 2 received from tool '/usr/bin/bash' 2022-10-25T19:18:19.6507493Z ##[debug]STDIO streams have closed for tool '/usr/bin/bash' 2022-10-25T19:18:19.6546007Z ##[error]ls: cannot access '/home/vsts/work/1/a/Output': No such file or directory 2022-10-25T19:18:19.6557766Z ##[debug]Processed: ##vso[task.issue type=error;]ls: cannot access '/home/vsts/work/1/a/Output': No such file or directory 2022-10-25T19:18:19.6558868Z ##[debug]task result: Failed 2022-10-25T19:18:19.6559820Z ##[error]Script has output to stderr. Failing as failOnStdErr is set to true. 2022-10-25T19:18:19.6560717Z ##[debug]Processed: ##vso[task.issue type=error;]Script has output to stderr. Failing as failOnStdErr is set to true. 2022-10-25T19:18:19.6562232Z ##[debug]Processed: ##vso[task.complete result=Failed;]Script has output to stderr. Failing as failOnStdErr is set to true. 2022-10-25T19:18:19.6563389Z ##[debug]which 'az' 2022-10-25T19:18:19.6564040Z ##[debug]found: '/usr/bin/az' 2022-10-25T19:18:19.6564648Z ##[debug]/usr/bin/az arg: account clear 2022-10-25T19:18:19.6565244Z ##[debug]/usr/bin/az arg: account clear 2022-10-25T19:18:19.6565822Z ##[debug]exec tool: /usr/bin/az 2022-10-25T19:18:19.6566396Z ##[debug]exec tool: /usr/bin/az 2022-10-25T19:18:19.6566942Z ##[debug]arguments: 2022-10-25T19:18:19.6567470Z ##[debug]arguments: 2022-10-25T19:18:19.6568009Z ##[debug] account 2022-10-25T19:18:19.6568535Z ##[debug] account 2022-10-25T19:18:19.6569056Z ##[debug] clear 2022-10-25T19:18:19.6569586Z ##[debug] clear 2022-10-25T19:18:19.6569941Z [command]/usr/bin/az account clear 2022-10-25T19:18:19.9663439Z ##[section]Finishing: Run extractor

hamedy commented 1 year ago

tried this and it didn't help sadly

mazer-rakham commented 1 year ago

This looks like a permission issue with vnets and subnets. I created a new APIM that is not behind a firewall, or subnets and vnets and the tool worked correctly.

guythetechie commented 1 year ago

@hamedy - thanks for re-running with the additional logging. I see what the problem is, but I'm not sure why it's occurring.

When the extractor runs, it extracts successfully to /home/vsts/work/1/s/Output/... Here are examples in the logs.

2022-10-25T19:18:19.6051118Z Exporting information for diagnostic applicationinsights...
2022-10-25T19:18:19.6107890Z Creating directory /home/vsts/work/1/s/Output/diagnostics/applicationinsights...
2022-10-25T19:18:19.6112307Z Creating directory /home/vsts/work/1/s/Output/diagnostics/azuremonitor...
2022-10-25T19:18:19.6114339Z Overwriting file /home/vsts/work/1/s/Output/diagnostics/azuremonitor/diagnosticInformation.json with JSON...
2022-10-25T19:18:19.6118482Z Overwriting file /home/vsts/work/1/s/Output/diagnostics/applicationinsights/diagnosticInformation.json with JSON...

However, when you run your ls command, it tries to pick up files from /home/vsts/work/1/a/Output. 

2022-10-25T19:18:19.6493920Z ls: cannot access '/home/vsts/work/1/a/Output': No such file or directory
2022-10-25T19:18:19.6497927Z Exit code is 2

I bet the Run publisher task resolves to the same path as your failing ls command. This is not where the extractor wrote its artifacts, so there's nothing there to publish. I'm not sure why $(Build.ArtifactStagingDirectory) in the pipeline changes from /home/vsts/1/s here https://github.com/Azure/apiops/blob/75dadd209a44409982ca78fcfd6590027697c3e1/tools/pipelines/run-extractor.yaml#L92 to /home/vsts/1/a here. https://github.com/Azure/apiops/blob/75dadd209a44409982ca78fcfd6590027697c3e1/tools/pipelines/run-extractor.yaml#L99

Will continue troubleshooting with the team, but at least we're in the right direction...

mazer-rakham commented 1 year ago

Yes the extractor is running successfully but there is no data in the extractor, so it is creating the artifacts-from-portal artifact, but it is completely empty, therefore later on the program cannot locate the items.

waelkdouh commented 1 year ago

This looks like a permission issue with vnets and subnets. I created a new APIM that is not behind a firewall, or subnets and vnets and the tool worked correctly.

@hamedy can you please check if you are facing the same problem that @mazer-rakham was facing as he was able to get around the issue.

waelkdouh commented 1 year ago

This looks like a permission issue with vnets and subnets. I created a new APIM that is not behind a firewall, or subnets and vnets and the tool worked correctly.

@mazer-rakham just to confirm. Your issue has been resolved correct?

mazer-rakham commented 1 year ago

More digging, enabled vnet and subnets and firewall and internal only on the APIM BUT under a new service principle and it is working also. I am thinking that someone saw a lot of traffic from ADO to APIM through the service principal and stopped it.... I am doing more research.

waelkdouh commented 1 year ago

@hamedy looking at @guythetechie response above it seems that in one location the pipeline is using s which is the source directory and in another location its using a which is the artifacts. As you can see in this link ADO letter meaning they mention the following: "On Hosted build agents "1" seems to be static, on a self hosted agent I have noticed this number is unique per pipeline". Can you please see if that is the culprit? Meaning are you using a self hosted build agent? Also @guythetechie suggested that you try the following to confirm that we are on the right path here: Instead of $(Build.ArtifactStagingDirectory), can you explicitly put "/home/vsts/1/a" in your pipeline. If everything works as expected, we can then try to figure out why it changes (self-hosted, etc).

mazer-rakham commented 1 year ago

This looks like a permission issue with vnets and subnets. I created a new APIM that is not behind a firewall, or subnets and vnets and the tool worked correctly.

@mazer-rakham just to confirm. Your issue has been resolved correct?

@waelkdouh not on the client's env, on my personal env. Everything the same except permissions so I am still trying to help here if need be.

waelkdouh commented 1 year ago

@hamedy looking at @guythetechie response above it seems that in one location the pipeline is using s which is the source directory and in another location its using a which is the artifacts. As you can see in this link ADO letter meaning they mention the following: "On Hosted build agents "1" seems to be static, on a self hosted agent I have noticed this number is unique per pipeline". Can you please see if that is the culprit? Meaning are you using a self hosted build agent?

Also @guythetechie suggested that you try the following to confirm that we are on the right path here:

Instead of $(Build.ArtifactStagingDirectory), can you explicitly put "/home/vsts/1/a" in your pipeline. If everything works as expected, we can then try to figure out why it changes (self-hosted, etc).

@mazer-rakham let me know if this suggestion works for you.

mazer-rakham commented 1 year ago

@waelkdouh OMG mine was so stupid so stupid. Someone had put this API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH in the variable group library AND defining it at pipeline run. It had cached it. I deleted the variable group and remade it WITHOUT the API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH and it works fine. Man oh man what a weird one. Thanks all

mazer-rakham commented 1 year ago

well dang now its back to the original issue. image

hamedy commented 1 year ago

ok, lots of weird stuff happening here on my side. I tried hard coding the path to 1/s and I no longer got any errors. Got some artifacts and a pull request. In the artifacts there was variables and logs and profucts, but no API's. After a couple of runs seemed like I wasn't getting any Pull Requests even though there was no errors.

now, I deleted all repos and grouped variables and started from scratch. tested a few times and everything is working, I get artifacts including API's and I get a pull request.

Another thing that also just started working, making the Pull Request to another repo, instead of the extractor repo. It used to give me permission errors even after giving permissions to my repo for all build service accounts. But now all of a sudden that works too.

I'll test more to see if this is working reliably. Thanks a lot everyone for the help so far.

waelkdouh commented 1 year ago

Glad to hear that. Sounds like you may be facing some permission issues that are unique to your environment. Please let us know when you finally get everything working and if there are any issues that we need to be aware of.

Please let us know if you manage to solve your issue without having to resort to hard coding the paths.

We will wait in your feedback before closing this issue.

hamedy commented 1 year ago

at first I think I forgot to run the publish extractor so it was the old version running with hard coded path. But I just ran publish extractor and everything still seems to be working fine. Will test more and report back in the coming days :)

hamedy commented 1 year ago

hi again :) after making some minor changes to the list of parameters in the extractor yaml file and running publish extractor, now run extractor fails again with the same error, no such file or directory.

hamedy commented 1 year ago

@waelkdouh can you please reopen the ticket?

waelkdouh commented 1 year ago

Hi Hamad. Just to be clear when you share issues here it doesn't mean you are opening a support ticket. It means you are raising potential issues with the tool. From what you have shared with us so far it seems the issues that you have been experiencing seem to be specific to your environment as you had it working at one point. Then it started failing again.

hamedy commented 1 year ago

here I think the bug is that if you have the output folder path in environment variables, with the same value as the parameter value, it breaks the code for some reason. The error went away when I removed the environment variable from the library. And I always had the same value from the beginning, so it's not a case of an old value being cached.

erydrn commented 8 months ago

I agree with @hamedy. I use v5.0.1. When I set API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH in the library and at the same time if I set this parameter before I run the pipeline, it keeps throwing an error "no such file or directory".

After I removed this parameter from the library as @hamedy suggested and set it before I ran the pipeline, the extractor worked without any problem and could extract the API from APIM. I think the development team should flag this!

Either this parameter should be removed as a required parameter to run the pipeline or this should not be defined in the library as it is mentioned in the guidance (https://azure.github.io/apiops/apiops/3-apimTools/apimtools-azdo-2-3-new.html)