Open rajivpatki opened 8 months ago
Are you still facing this? Tempted to close as stale, assuming you figured something out. I know we're using a ton of google sheets with credentials at Preset, so I suspect there's something in the credentials causing a problem (maybe a character like an @ or something), or this was an intermittent bug that came and went. As a side note, you should set a new secret key, too ;)
Let's see if our shiny new bot has more advice:
@dosu-bot
It looks like the issue you're experiencing with Superset and Google Sheets integration, particularly after adding service account credentials, might be multifaceted. The error message indicating that the URL could not be identified, despite your detailed troubleshooting, suggests a deeper issue possibly related to how Superset is handling or parsing the service account credentials.
Here are some refined steps and considerations to help you troubleshoot and hopefully resolve the issue:
Ensure Correct Service Account Credentials Format: Double-check that the JSON credentials for your service account are correctly formatted. This includes making sure that all necessary fields are present and that special characters, especially in the private_key
, are properly escaped. Incorrect formatting or missing fields can lead to authentication issues.
Verify Google Sheet Sharing Settings: Make sure the Google Sheet you're trying to access is shared with the client_email
found in your service account credentials. This step is crucial for granting the necessary access permissions to your service account.
Review Superset Configuration: Go through your superset_config.py
file again to confirm that everything is set up correctly for Google Sheets integration. This includes checking the FEATURE_FLAGS
and any other configurations that might affect how external databases are connected.
Correct Use of Encrypted Extra: In the database connection UI, ensure that the Encrypted Extra
field accurately contains your service account credentials as a JSON string. This often involves making sure the JSON is correctly stringified and that no formatting issues are introduced.
Pay Attention to Error Messages and Logs: The error messages and logs from Superset can provide valuable insights into what might be going wrong. Since you've encountered specific errors related to the URL identification and potential issues with \n
characters in your JSON, it's worth revisiting these messages for any clues they might offer.
Check Superset Version Compatibility: Verify that your current version of Superset (3.0.2) fully supports the features and methods you're using for Google Sheets integration. If there have been updates or bug fixes in more recent versions that address your issue, upgrading might be necessary.
Given the steps you've already taken, including attempts to modify the gsheets.py
script to handle \n
characters, it's clear that the problem might not be straightforward. If these suggestions don't lead to a resolution, it could indicate a more specific issue with the gsheets.py
script or a unique aspect of your setup. Ensuring that all configurations and credentials are correct and looking into any potential updates or fixes in newer versions of Superset could be key to resolving this.
To continue the conversation, mention @dosu.
Are you still facing this? Tempted to close as stale, assuming you figured something out. I know we're using a ton of google sheets with credentials at Preset, so I suspect there's something in the credentials causing a problem (maybe a character like an @ or something), or this was an intermittent bug that came and went. As a side note, you should set a new secret key, too ;)
Let's see if our shiny new bot has more advice:
@dosu-bot
This issue still exists for me in Superset 3.1.2. Our "workaround" is not ideal, we have to schedule a script to regularly update another database with the googlesheet contents
If I don't provide any credentials.json for a pulicly shared sheet, the connection works fine. As soon as credentials are specified I get the error "The URL could not be identified. Please check for typos and make sure that ‘Type of Google Sheets allowed’ selection matches the input." which is the same error that I get for private sheets.
To further address the persistent issue you're encountering with connecting Google Sheets to Superset 3.1.2 using service account credentials, even after ensuring the basic setup and configurations are correct, let's delve into a more detailed approach:
Ensure the Google Sheets URL is Correct: The format should strictly follow https://docs.google.com/spreadsheets/d/[SPREADSHEET_ID]/edit
. Any deviation from this format could lead to connection issues.
Verify Service Account Permissions: It's crucial that the Google Sheet is explicitly shared with the client_email
associated with your service account. This step is often overlooked but is essential for granting access.
Check the Format of Your Service Account Credentials: The JSON credentials must be correctly formatted. This includes ensuring that the private_key
within the JSON is properly escaped. Given that you've confirmed these credentials work with a Python script, this might not be the issue, but it's worth re-verifying, especially after any modifications.
Catalog Configuration: In Superset, ensure that there's a correct mapping between the table names you're using and the URLs of the Google Sheets. This might involve reviewing the configuration files or settings within Superset that handle external database connections.
Match the Type of Google Sheets Allowed: The error message you're encountering suggests a mismatch in the settings regarding the type of Google Sheets allowed. Double-check that your Superset configuration aligns with the permissions you've set up in the Google Cloud Platform and within Superset's database connection settings.
OAuth2 and Impersonation Settings: If your setup involves OAuth2 or impersonating a user account, ensure that all the necessary scopes and parameters are correctly configured. This includes verifying that the service account has the appropriate permissions to access the Google Sheets API.
If after following these steps, the issue remains unresolved, consider exploring these additional solutions:
Change the Google Sheet's Locale: Some users have found success by setting their Google Sheet's country setting to the United States. This might help if there's a locale-specific issue affecting the connection.
Review Superset's Documentation for Private Sheets: There might be additional steps or configurations required for connecting privately shared Google Sheets that are not immediately obvious. Superset's official documentation or community forums may have updated guides or solutions based on the latest versions.
Given the complexity of the issue and the steps you've already attempted, it's clear that a solution may require a combination of verifying configurations, permissions, and perhaps even looking into recent updates or changes in Superset's handling of Google Sheets connections. If these suggestions do not lead to a resolution, it could be beneficial to reach out to the Superset community or review open issues in the Superset GitHub repository for any similar cases and potential fixes.
To continue the conversation, mention @dosu.
@dosu
I also tested using a public google sheet that was also shared with the service account email address. The connection works fine without the credentials. If I supply any credentials, the error that I mentioned earlier shows up.
To address the issue with connecting Google Sheets to Superset 3.1.2 using service account credentials, consider the following targeted steps:
Ensure Correct Google Sheets URL Format: The URL should strictly follow the format https://docs.google.com/spreadsheets/d/[SPREADSHEET_ID]/edit
. Any deviation could lead to connection issues.
Verify Service Account Permissions: Confirm that the Google Sheet is shared with the client_email
from your service account credentials, granting at least viewer access.
Check Service Account Credentials Format: Ensure the JSON credentials are correctly formatted, especially the private_key
, and contain all required fields.
Review 'Type of Google Sheets allowed' Selection: Double-check that your Superset configuration aligns with the permissions set up in Google Cloud Platform and Superset's database connection settings.
Inspect Encrypted Credentials Handling: If using encrypted credentials, ensure they are correctly decrypted and valid.
Enable Google Sheets API: Verify that the Google Sheets API is enabled for the project associated with the service account.
Update Superset and Dependencies: Make sure your Superset installation and its dependencies are up to date.
Debug Connection Attempt: Utilize logging to identify where the connection process might be failing.
By following these steps and ensuring all configurations and permissions are correctly set, you should be able to resolve the connection error. Additionally, consider changing the Google Sheet's locale settings or ensuring data is in text format as potential solutions.
To continue the conversation, mention @dosu.
A clear and concise description of what the bug is.
How to reproduce the bug
Expected results
con
being created -- Logs cannot distinguish between invalid input file, credentials failed, failure to read sheets or failure to create a connectionActual results
On adding the credentials, you see on the interface The URL could not be identified. Please check for typos and make sure that ‘Type of Google Sheets allowed’ selection matches the input![image](https://github.com/apache/superset/assets/54361216/05d7f955-2dba-45c2-86dd-88abbf8ff4cf)
In superset logs you see:
Environment
browser type and version: Microsoft Edge Version 120.0.2210.91 (Official build) (64-bit) superset version: 3.0.2 from official docker image any feature flags active:
Checklist
Make sure to follow these steps before submitting your issue - thank you!
Additional context
superset/db_engine_specs/gsheets.py
json.loads
on this json does not work because it cannot read\n
charactersgsheets.py
withstrict = False
forjson.loads
but that did not work on Superset while it works on python: refer this stackoverflow answer\n
with\\n
but that too throws the same error while it works on python: refer to this stackoverflow answer