Generic connections showing different results in local when compared to astro

[rawwar]

Describe the bug

I created a Connection of type "Google Cloud Platform" using "Generic" on Astro Cloud UI. Following are the steps to replicate the creation

1. Open Astronomer Cloud UI
2. Go to "Environment"
3. Create New Connection
4. Choose "Generic" connection type
5. Fill in the details and input "TYPE" with "google_cloud_platform" and "EXTRA" with the gcp service account JSON value(One I used is a dummy one and no longer exists. But, I did use the original keyfile while replicating the issue) and create the connection.

{
  "type": "service_account",
  "project_id": "TEST",
  "private_key_id": "Private key",
  "private_key": "-----BEGIN PRIVATE KEY-----\nKeyAgEAAoIBAQDnxjrrvjt806XB\nm4ZAnXs5TA3MhwqsqyClzZQhxBHzGJJKn1zdNtB9eIYrC4PNxQRCJPKw5WmyyCy6\ncRwTyilSmWlPKSdD7CLiK6JFZV8mZqVpa86CvTr12GYST590J8P9zvc9AaMldCDM\nYFmruxhTV+pgYteZwPX7iU2mPpLz9Hom/9YlsQfvZ07b2BlPVXUkN0pGqwoiCqqQ\nOUJATpJ8HqPLPdGxJ4RXHFNbshBXpBCrAUfP0javOzBzGsDWXiAEFhXenFTqDc8Y\nUKObtKC/I9nkP98L9FVX66K+rRJrqy6xWPfLNlop1vHvWWKrPqDUaQfpzbmOgYlo\nSZ+wkW81AgMBAAECggEAMs1ritLGvUSy9238jls2cEewWp1bBJ/55v3Zs1ED0jIX\nls25kC98oZPybKr4d/jwRwNMQuP2OB7SVweF7PyJ9DX2NEjD94ph1wtTZTykD4m+\nRg5AVL3fl5WYQfk77TzuI8rivHeIaCQ9MC8KWK8I1PPXRstDkMsP39OntoHOujP4\nU7gnFQuZEvuP2jf7ajts2gwlPTTUkp55+lASMil/JV+6OBaCel0ZAjR7hkTNqS/O\nbqisvmhCHFcDXlZOJgk2AevAjSgtYBoYZmLNZizRL2qlrRR2C04tHQaRKlkNsjC1\nNxKWJoF8AlQ2CeItu11KLC4gm1PbUI/p7n3ke662vQKBgQD+aOdMYAN6pIvp+fFP\n98PFyneCsHtP1BZBcx89dW2wJpoPnlJOXKCdNNmEqWbvIwdbakqBqIKodDrs+JLF\nc+tBXsg8KU++WLNrbmPgvTHEhvV/uY6nPesRAsPRnbAEjjT4DDWUH8cMsTouQruu\n0cgnj1YqcaP0tT0z7d5FRG65KwKBgQDpORs3Aes/FUaFBIx1OQI90mFSnt0zw9uz\nLpDAaKdw9WSmkfPca4qjf1TvsA9b6IsbiZ4+use/569qG8hBBff5hH/v5acS/Vu8\nGe1Dl2IiYH7QE3N7x1Tf1f1qyKnLETmPoTQMTxw2JYtYumxqjzhALjQxwUrsLCGF\nP9Okh7OJHwKBgQCpdylzh5VpsYf9oF9s/EatTxPSu0o4v7bffBb4MmuRZfVDqBt9\nVVgrv9OQ+eK2Mqgo7aGZwqzeFyZVSbNC0deIoOFSqdYTnjRNwKfJQV2QYrHPlcP/\nhVA68ZOM4rQoJK4F+OQLuis2OiveOteYOOdR6rnj5fToAX42RCHGFDKJJwKBgQDG\nQXxIC4KOEqjq5I3+DCNCqHeBA3koywMVU61LUuub047nJkU2B9lH2TM7ud+kko7G\nLvttedvZ37q3KbzWwZsl6iPTq9aWBvgXc7bE8YhDjuHeQdq2yyD0UnVldAUakZ7r\n4O5c1Fc4FDxbhcYp42u+60gFT7o//9fS7Qod4BmXowKBgBYJl42Lf2ZffrcV5Bvo\nXZbQphlobCGt/CaQEvEMBen3mnj/56w6kEC7WGLOVu2c66Lus6cn5pEdSFzdn5YZ\n+118PserBiEmtvA28b2xiAOWL01C+2rAYBjqkFV3Yf2K3qGAXsX/yYnTZKzREavH\n07er62ieqgiasdBdY9WdDHoY\n-----END PRIVATE KEY-----\n",
  "client_email": "test_email.iam.gserviceaccount.com",
  "client_id": "09213890182301283",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/t4wersvc%40cre-sandbox-349214.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

6. Once created, link it with a deployment.

On Astro:

Once the connection is created and linked with the deployment, Exec into the worker pod and run airflow connections get gcp_test_2. This will return the following.

In Local:

I followed the following article for running a local deployment - Link

Commands to start the env:

1. astro login  #complete the login
2. astro config set -g disable_env_objects false
3. astro dev start --deployment-id <Deployment id which has the gcp_test_2 connection linked>

Now, exec into a container(I've exec'ed into scheduler) and run airflow connections get gcp_test_2 Command returns the following:

extra_dejson is empty in local. Also, we can see a warning saying that the astro managed secrets backend is disabled`

I have tested the above for simple JSON data in extra field like {"test":"hello"} and for this case, I can see similar results in local and in astro env. But, when giving Google Cloud's keyfile json data, this issue is happening.

What CLI Version did you experience this bug?

Astro CLI Version: 1.24.1

This CLI bug is related to which Astronomer Platform?

• [x] Astro
• [ ] Software
• [ ] None/Unknown

What Operating System is the above CLI installed on? IOS

[jeremybeard]

@rawwar Has environment secrets fetching been enabled on the organization, per these docs?

[rawwar]

Member

yes, its enabled. I think there's some issue with parsing the extra field. Otherwise, using a simple JSON like {'test':'hello_world'} in extra wouldn't work.

[kushalmalani]

The fix was released in 1.25.0