search

HodorNV / ALOps

ALOps
58 stars 24 forks source link

Performance ALOpsOpenAPI #501

Closed Arthurvdv closed 2 years ago

Arthurvdv commented 2 years ago

Describe the bug The idea was to generate the OpenAPI documentation on every pull-request. This PTE with ±60 API pages takes more then 60minutes to run the ALOpsOpenAPI-step isn't workable. Just a wild guess here: Maybe the step is caught in a infinite loop?

The OpenAPI-folder in the artifacts folder on the buildserver is empty.

the used yaml please provide the yaml that you used. It helps you put the yaml like this: 

steps:
- task: ALOpsOpenAPI@1
  displayName: 'Generate OpenAPI documentation'
  condition: and(succeeded(), ${{ parameters.execute }})
  continueOnError: true
  inputs:
    usedocker: true
    api_endpoint: 'https://$(ALOPS_DOCKER_HOSTNAME):7048/bc/api'
    authentication: 'Basic'
    username: '$(ALOPS_DOCKER_ADMINUSER)'
    password: '$(ALOPS_DOCKER_ADMINPASSWORD)'
    publish_artifacts: true
    discover: true
    api_exclude_routes: |
      beta
      v1.0
      v2.0
      microsoft/admin/beta
      microsoft/automation/beta
      microsoft/automation/v1.0
      microsoft/automation/v2.0
      microsoft/cloudMigration/v1.0
      microsoft/powerbi/v2.0
      microsoft/runtime/beta
      microsoft/vatGroup/v1.0
    export_yaml: true
    export_json: false

the output Also the complete output is necessary for us to see what is going on. Also use backtics:

2022-05-03T07:11:15.7610500Z ##[section]Starting: Generate OpenAPI documentation
2022-05-03T07:11:15.7706865Z ==============================================================================
2022-05-03T07:11:15.7707072Z Task         : ALOps OpenAPI
2022-05-03T07:11:15.7707258Z Description  : Generate OpenAPI descriptions from Business Central API's
2022-05-03T07:11:15.7707431Z Version      : 1.448.2984
2022-05-03T07:11:15.7707559Z Author       : Hodor
2022-05-03T07:11:15.7707733Z Help         : Generate OpenAPI descriptions from Business Central API's.
2022-05-03T07:11:15.7707949Z ==============================================================================
2022-05-03T07:11:16.9577421Z *** Validate configuration
2022-05-03T07:11:17.1774339Z *** Task Inputs:
2022-05-03T07:11:17.1806914Z 
2022-05-03T07:11:17.1872647Z name                                                                                                              value
2022-05-03T07:11:17.1874533Z ----                                                                                                              -----
2022-05-03T07:11:17.1875141Z usedocker                                                                                                          True
2022-05-03T07:11:17.1876952Z fixed_tag                                                                                                              
2022-05-03T07:11:17.1877638Z api_endpoint                                                                           https://ef547fcd24d2:7048/bc/api
2022-05-03T07:11:17.1877917Z authentication                                                                                                    Basic
2022-05-03T07:11:17.1879696Z azure_tenant_id                                                                                                        
2022-05-03T07:11:17.1880320Z azure_app_client_id                                                                                                    
2022-05-03T07:11:17.1881641Z azure_app_client_secret                                                                                                
2022-05-03T07:11:17.1882254Z username                                                                                                          admin
2022-05-03T07:11:17.1884016Z password                                                                                                       Fixo5517
2022-05-03T07:11:17.1884338Z publish_artifacts                                                                                                  True
2022-05-03T07:11:17.1884571Z discover                                                                                                           True
2022-05-03T07:11:17.1886183Z api_routes                                                                                                             
2022-05-03T07:11:17.1886772Z api_exclude_routes                                                                                              beta...
2022-05-03T07:11:17.1887082Z inspect_paths                                                                                                     False
2022-05-03T07:11:17.1889282Z api_servers                                                                                                            
2022-05-03T07:11:17.1889910Z export_edmx                                                                                                       False
2022-05-03T07:11:17.1891394Z export_yaml                                                                                                        True
2022-05-03T07:11:17.1891659Z export_json                                                                                                       False
2022-05-03T07:11:17.1891773Z 
2022-05-03T07:11:17.1904390Z 
2022-05-03T07:11:17.1909376Z 
2022-05-03T07:11:17.2013271Z *** For documentation, please visit   : https://www.alops.be/documentation
2022-05-03T07:11:17.2015876Z 
2022-05-03T07:11:17.6308316Z *** ALOps License:
2022-05-03T07:11:17.6320765Z   * Licensed To: Van Roey (Collection License)
2022-05-03T07:11:17.6323192Z 
2022-05-03T07:11:17.6386896Z *** Imported required PS-Functions
2022-05-03T07:11:18.5029641Z *** Parse Task Parameters
2022-05-03T07:11:18.5135371Z *** Exclude Routes:
2022-05-03T07:11:18.5198582Z   * beta
2022-05-03T07:11:18.5210587Z   * v1.0
2022-05-03T07:11:18.5215191Z   * v2.0
2022-05-03T07:11:18.5219040Z   * microsoft/admin/beta
2022-05-03T07:11:18.5222507Z   * microsoft/automation/beta
2022-05-03T07:11:18.5226696Z   * microsoft/automation/v1.0
2022-05-03T07:11:18.5232420Z   * microsoft/automation/v2.0
2022-05-03T07:11:18.5235811Z   * microsoft/cloudmigration/v1.0
2022-05-03T07:11:18.5239525Z   * microsoft/powerbi/v2.0
2022-05-03T07:11:18.5243224Z   * microsoft/runtime/beta
2022-05-03T07:11:18.5246880Z   * microsoft/vatgroup/v1.0
2022-05-03T07:11:18.7092851Z *** Initiate Docker Session
2022-05-03T07:11:18.9070729Z *** Set Docker Container ErrorActionPreference = Stop
2022-05-03T07:11:19.9833512Z *** Creating API Connector with endpoint [https://ef547fcd24d2:7048/bc/api]
2022-05-03T07:11:20.0002886Z *** API Authentication: Using Basic Authentication [admin]
2022-05-03T07:11:20.0042167Z *** List Companies
2022-05-03T07:11:20.0127287Z ##[command]Invoke-RestMethod -Method Get -Uri 'https://ef547fcd24d2:7048/bc/api/microsoft/automation/v1.0/companies'
2022-05-03T07:11:30.6629195Z 
2022-05-03T07:11:30.6676859Z id                                   name       displayName
2022-05-03T07:11:30.6678833Z --                                   ----       -----------
2022-05-03T07:11:30.6680507Z 8df900cc-39c9-ec11-bb91-000d3a39a4f0 CRONUS BE             
2022-05-03T07:11:30.6682187Z af2f98d9-39c9-ec11-bb91-000d3a39a4f0 My Company            
2022-05-03T07:11:30.6682326Z 
2022-05-03T07:11:30.6683290Z 
2022-05-03T07:11:30.6722376Z *** Selected Company: CRONUS BE [8df900cc-39c9-ec11-bb91-000d3a39a4f0]
2022-05-03T07:11:32.8379887Z *** Loading Assemblies:
2022-05-03T07:11:32.8381492Z   * Microsoft.OData.Edm, Version=7.6.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
2022-05-03T07:11:32.8383271Z   * Microsoft.OpenApi, Version=1.2.2.0, Culture=neutral, PublicKeyToken=3f5743946376f042
2022-05-03T07:11:32.8385266Z   * Microsoft.OpenApi.OData.Reader, Version=1.0.9.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
2022-05-03T07:11:32.8387147Z *** Discover Routes
2022-05-03T07:11:35.3427883Z *** Exclude Routes
2022-05-03T07:11:35.3463911Z   * beta
2022-05-03T07:11:35.3476354Z   * v1.0
2022-05-03T07:11:35.3484669Z   * v2.0
2022-05-03T07:11:35.3492148Z   * microsoft/admin/beta
2022-05-03T07:11:35.3500008Z   * microsoft/automation/beta
2022-05-03T07:11:35.3508254Z   * microsoft/automation/v1.0
2022-05-03T07:11:35.3516901Z   * microsoft/automation/v2.0
2022-05-03T07:11:35.3523009Z   * microsoft/cloudmigration/v1.0
2022-05-03T07:11:35.3529819Z   * microsoft/powerbi/v2.0
2022-05-03T07:11:35.3536515Z   * microsoft/runtime/beta
2022-05-03T07:11:35.3543458Z   * microsoft/vatgroup/v1.0
2022-05-03T07:11:35.3550246Z *** Routes to Convert:
2022-05-03T07:11:35.3557139Z 
2022-05-03T07:11:35.3569697Z Endpoint                         Route                   OpenAPI ConvertDuration
2022-05-03T07:11:35.3571932Z --------                         -----                   ------- ---------------
2022-05-03T07:11:35.3573645Z https://ef547fcd24d2:7048/bc/api vanRoey/automation/v2.0                       0
2022-05-03T07:11:35.3574959Z https://ef547fcd24d2:7048/bc/api vanRoey/powerBI/v2.0                          0
2022-05-03T07:11:35.3575125Z 
2022-05-03T07:11:35.3575744Z 
2022-05-03T07:11:35.3582973Z *** Convert Route: vanRoey/automation/v2.0
2022-05-03T07:11:35.3847780Z ##[command]Invoke-RestMethod -Method Get -Uri 'https://ef547fcd24d2:7048/bc/api/vanRoey/automation/v2.0/$metadata?$schemaversion=2.0'
2022-05-03T07:11:37.7909614Z *** Parse EDMX Content
2022-05-03T07:11:37.9496017Z *** Convert To OpenAPI
2022-05-03T08:05:19.5212503Z ##[error]The operation was canceled.
2022-05-03T08:05:19.5248501Z ##[section]Finishing: Generate OpenAPI documentation

Expected behavior Dependent of the amount of API-objects the time to generate the OpenAPI documentation could vary. My example PTE hasn't extraordinary much API endpoints. I would expect this step to run for a few minutes tops.

Screenshots image

AdminHodor commented 2 years ago

Dear @Arthurvdv ,

Could you supply us the EDMX (via Email if you want to keep it private) of the API thats fails to convert ? That would allow us quicker debug any possible issue.

Kind regards

Arthurvdv commented 2 years ago

@AdminHodor, thanks! I've e-mailed you the file(s).

waldo1001 commented 2 years ago

Thanks. We are looking into it. Dear lord ;-). It's a big file 😱.
Not sure if we'll be able to resolve it though ... we're considering our options..

waldo1001 commented 2 years ago

The problem is the size of the file, and the number of paths it's causing. And the library we're using to go from edmx to openapi does not take a depth decently .. so it will produce ALL combinations of paths. Having such a huge API, this is an exploding problem.

I'd recommend to keep the API as small as possible .. that's all I can say for the time being .. we're not sure we'll find a solution, to be honest.. .

waldo1001 commented 2 years ago

We deeply investigated. Since we use an OpenSource library, and the combination with how busines central sets up the multi-level routing .. we come into an endless number of navigation paths. With big APIs (like the default) this is just insane and breaks..

We're going to have to pause this for now.. .

Recommendation: only for custom APIs, only when you have small amount of APIs available in a group.

Arthurvdv commented 2 years ago

Thank you for investigating. Unfortunately to hear its not solvable at this moment. Was looking forward to enable this as default on our projects and add this shippable artifact next to the .app file(s).

To be honest it's not a deal breaker at this moment, but it would be awesome to use this feature in the future.

waldo1001 commented 2 years ago

We're not giving up yet - the OpenAPI library has some misteries we need to uncover before we give up ;-)

AdminHodor commented 2 years ago

Dear @Arthurvdv,

The fix was released today in ALOps v1.450. Could you please validate? Downgrading the OpenAPI.OData component from v1.0.9 to v1.0.8 seemed to have done the trick.

Kind regards,

waldo1001 commented 2 years ago

@Arthurvdv - were you able to test this?

Arthurvdv commented 2 years ago

I’m currently out-of-the-office. I’l be back beginning of August and make sure to test this again.

Arthurvdv commented 2 years ago

Awesome, down to merely 27 seconds!

image