search

astronomer / astro-cli

CLI that makes it easy to create, test and deploy Airflow DAGs to Astronomer
https://www.astronomer.io
Other
355 stars 70 forks source link

Updating worker concurrency doesn't reflect in the response or UI #1267

Closed shri-astro closed 1 year ago

shri-astro commented 1 year ago

While attempting to update the worker-queue's concurrency through astro deployment worker-queue update command, using the flag --concurrency int to alter the concurrency of a given astro machine doesn't change the concurrency value for the updated worker queue. However, the astromachine gets updated successfully.

shrividyahegde@Shrividyas-MacBook-Pro Documents % astro deployment worker-queue update --concurrency 30 
Select a Deployment
 #     DEPLOYMENT NAME                       RELEASE NAME                     DEPLOYMENT ID                  DAG DEPLOY ENABLED     
 1     gcp_test_standard                     planetoidal-gegenschein-1614     clix117rh103281jxa9sgyorxg     false                  
 2     qwerty                                barren-parallax-1415             clix15tmm117561iyyf65j7ghx     false                  
 3     gcp_test_standard_through_inspect     ultraviolet-ionosphere-7533      clix47lns235701iz1ddgrjtka     false                  
 4     gcp_standard                          newtonian-ionization-1282        clix5x8nv342781jxabusoal5z     false                  

> 4
No worker type was specified. Select the worker type to use
 #     WORKER TYPE     CPU        Memory     
 1     a5              1 vCPU     2Gi        
 2     a10             2 vCPU     4Gi        
 3     a20             4 vCPU     8Gi        

> 3

 #     WORKER QUEUE     ISDEFAULT     ID                             
 1     default          true          clix5x8nv342801jxa5p5nrvd1     
 2     non-default      false         clix6dvkn357911jxa8p6jcif0     

> 2

Are you sure you want to update the non-default worker queue? If there are any tasks in your DAGs assigned to this worker queue, the tasks might get stuck in a queued state and fail to execute (y/n) y
worker queue non-default for gcp_standard in clix0m97m001i01oiqr98vqb5 workspace updated
shrividyahegde@Shrividyas-MacBook-Pro Documents % astro deployment inspect                             
Select a Deployment
 #     DEPLOYMENT NAME                       RELEASE NAME                     DEPLOYMENT ID                  DAG DEPLOY ENABLED     
 1     gcp_test_standard                     planetoidal-gegenschein-1614     clix117rh103281jxa9sgyorxg     false                  
 2     qwerty                                barren-parallax-1415             clix15tmm117561iyyf65j7ghx     false                  
 3     gcp_test_standard_through_inspect     ultraviolet-ionosphere-7533      clix47lns235701iz1ddgrjtka     false                  
 4     gcp_standard                          newtonian-ionization-1282        clix5x8nv342781jxabusoal5z     false                  

> 4
deployment:
    configuration:
        name: gcp_standard
        description: ""
        runtime_version: 8.4.0
        dag_deploy_enabled: false
        ci_cd_enforcement: true
        scheduler_size: small
        is_high_availability: false
        executor: CeleryExecutor
        scheduler_au: 10
        scheduler_count: 1
        cluster_name: us-central1
        workspace_name: Test Automated Workspace
        deployment_type: HOSTED_SHARED
        cloud_provider: gcp
        region: us-central1
    worker_queues:
        - name: default
          max_worker_count: 10
          min_worker_count: 0
          worker_concurrency: 20
          worker_type: a20
        - name: non-default
          max_worker_count: 10
          min_worker_count: 0
          worker_concurrency: 20
          worker_type: a20
    metadata:
        deployment_id: clix5x8nv342781jxabusoal5z
        workspace_id: clix0m97m001i01oiqr98vqb5
        cluster_id: us-central1
        release_name: N/A
        airflow_version: 2.6.1
        current_tag: 8.4.0
        status: UNKNOWN
        created_at: 2023-06-15T13:14:20.827Z
        updated_at: 2023-06-15T13:28:15.54Z
        deployment_url: cloud.astronomer-dev.io/clix0m97m001i01oiqr98vqb5/deployments/clix5x8nv342781jxabusoal5z/analytics
        webserver_url: org-shri-hosted-dedicated-1.astronomer-dev.run/dusoal5z
        workload_identity: astro-newtonian-ionization-128@astro-gcp-data-plane-dev.iam.gserviceaccount.com

What CLI Version did you experience this bug? 1.16.1

This CLI bug is related to which Astronomer Platform?

📸 Screenshots

Screenshot 2023-06-15 at 7 01 34 PM
kushalmalani commented 1 year ago

@shri-astro To test this on latest main

shri-astro commented 1 year ago

Able to set the worker concurrency now but along with a few observations:

  1. The initial range mentioned for worker concurrency is misleading 
    
shrividyahegde@Shrividyas-MacBook-Pro test_dep % ./astro deployment worker-queue update --concurrency 90 
Select a Deployment
#     DEPLOYMENT NAME     RELEASE NAME                DEPLOYMENT ID                   DAG DEPLOY ENABLED     
1     test                galactic-eclipse-4067       clj72yh7w1174401u2kvvqgab9u     false                  
2     nknk                sidereal-protostar-7638     clj73oa4y1355441uvihu17g8o5     false

1 No worker type was specified. Select the worker type to use

WORKER TYPE CPU Memory

1 a5 1 vCPU 2Gi
2 a10 2 vCPU 4Gi
3 a20 4 vCPU 8Gi

3

WORKER QUEUE ISDEFAULT ID

1 default true clj72yh7w1174431u2kpouibanf

1 Error: worker queue option is invalid: worker concurrency must be between 1 and 64 shrividyahegde@Shrividyas-MacBook-Pro test_dep % ./astro deployment worker-queue update --concurrency 64 Select a Deployment

DEPLOYMENT NAME RELEASE NAME DEPLOYMENT ID DAG DEPLOY ENABLED

1 test galactic-eclipse-4067 clj72yh7w1174401u2kvvqgab9u false
2 nknk sidereal-protostar-7638 clj73oa4y1355441uvihu17g8o5 false

1 No worker type was specified. Select the worker type to use

WORKER TYPE CPU Memory

1 a5 1 vCPU 2Gi
2 a10 2 vCPU 4Gi
3 a20 4 vCPU 8Gi

3

WORKER QUEUE ISDEFAULT ID

1 default true clj72yh7w1174431u2kpouibanf

1

Are you sure you want to update the default worker queue? If there are any tasks in your DAGs assigned to this worker queue, the tasks might get stuck in a queued state and fail to execute (y/n) y Error: The worker concurrency must be between 1 and 60 (Requested: 64).

kushalmalani commented 1 year ago

@shri-astro Can you take another look at this when you can? You would need to pull latest

shri-astro commented 1 year ago

On testing the latest main version,it was observed that for a5 the user is not able to set a concurrency within 1-5, the lower limit starts as 5

Error: worker queue option is invalid: worker concurrency must be between 5 and 15

UI allows the update from 1-15.

Similarly for a20:

Error: worker queue option is invalid: worker concurrency must be between 20 and 60

and A10:

Error: worker queue option is invalid: worker concurrency must be between 10 and 30

setting the upper limits like , 15 as concurrency for a5, 30 for a10 and 60 for a20 is working fine.

kushalmalani commented 1 year ago

@shri-astro Fixed this in latest main

shri-astro commented 1 year ago

Retested and able to update the worker concurrency but the messages displayed while trying outer range of concurrency for a given worker type still displays a wrong lower boundary value. i.e, updating a a5 by worker concurrency of 120 , displays the below message:

Error: worker queue option is invalid: worker concurrency must be between 5 and 15

However, user is able to update the worker concurrency in this case to 2 as well.(any value less than 5)

shri-astro commented 1 year ago

Interesting case being updating of concurrency of a given worker queue to 0 displays a success message but functionally doesn't change anything in the deployment:

shrividyahegde@Shrividyas-MacBook-Pro test_dep % ./astro deployment worker-queue update --concurrency 0  
Select a Deployment
 #     DEPLOYMENT NAME     RELEASE NAME                 DEPLOYMENT ID                   DAG DEPLOY ENABLED     
 1     test                true-synodic-0298            cljebq2bf2921391uwqyxrdyx8n     true                   
 2     test health         empty-scintillation-8984     cljeitgb8100681uwwbnwpi5tp      false                  
 3     test111             planetary-gyroscope-4233     cljeiv9uo104381uwwcja9z7a2      true                   

> 2
No worker type was specified. Select the worker type to use
 #     WORKER TYPE     CPU        Memory     
 1     a5              1 vCPU     2Gi        
 2     a10             2 vCPU     4Gi        
 3     a20             4 vCPU     8Gi        
 4     a40             8 vCPU     16Gi       

> 2

 #     WORKER QUEUE     ISDEFAULT     ID                             
 1     default          true          cljeitgb8100721uwwum3nvwlj     

> 1

Are you sure you want to update the default worker queue? If there are any tasks in your DAGs assigned to this worker queue, the tasks might get stuck in a queued state and fail to execute (y/n) y
worker queue default for test health in clix0m97m001i01oiqr98vqb5 workspace updated
shrividyahegde@Shrividyas-MacBook-Pro test_dep % ./astro deployment inspect                              
Select a Deployment
 #     DEPLOYMENT NAME     RELEASE NAME                 DEPLOYMENT ID                   DAG DEPLOY ENABLED     
 1     test                true-synodic-0298            cljebq2bf2921391uwqyxrdyx8n     true                   
 2     test health         empty-scintillation-8984     cljeitgb8100681uwwbnwpi5tp      false                  
 3     test111             planetary-gyroscope-4233     cljeiv9uo104381uwwcja9z7a2      true                   

> 2
deployment:
    configuration:
        name: test health
        description: ""
        runtime_version: 8.6.0
        dag_deploy_enabled: false
        ci_cd_enforcement: false
        scheduler_size: small
        is_high_availability: false
        executor: CeleryExecutor
        scheduler_au: 10
        scheduler_count: 1
        cluster_name: us-central1
        workspace_name: Test Automated Workspace
        deployment_type: HOSTED_SHARED
        cloud_provider: gcp
        region: us-central1
    worker_queues:
        - name: default
          max_worker_count: 10
          min_worker_count: 0
          worker_concurrency: 3
          worker_type: a10
    metadata:
        deployment_id: cljeitgb8100681uwwbnwpi5tp
        workspace_id: clix0m97m001i01oiqr98vqb5
        cluster_id: us-central1
        release_name: N/A
        airflow_version: 2.6.2
        current_tag: 8.6.0
        status: DEPLOYING
        created_at: 2023-06-27T16:47:24.116Z
        updated_at: 2023-06-27T16:54:18.648Z
        deployment_url: cloud.astronomer-dev.io/clix0m97m001i01oiqr98vqb5/deployments/cljeitgb8100681uwwbnwpi5tp/analytics
        webserver_url: org-shri-hosted-dedicated-1.astronomer-dev.run/dnwpi5tp
        workload_identity: astro-empty-scintillation-8984@astro-gcp-data-plane-dev.iam.gserviceaccount.com
shri-astro commented 1 year ago

Resolved. retest results:

Error: Worker concurrency cannot be 0. Minimum value starts from 1
Error: worker queue option is invalid: worker concurrency must be between 1 and 15
Error: worker queue option is invalid: worker concurrency must be between 1 and 30
Error: worker queue option is invalid: worker concurrency must be between 1 and 60
Error: worker queue option is invalid: worker concurrency must be between 1 and 120