Closed susodapop closed 3 years ago
@arikfr I had a go at testing some of this and there are a large number of issues that came up when trying to migrate. Their were some small formatting issues and incorrect calls to get/post, but the biggest one was that it didn't seem to recognize pending invitations to users as real users. So when you try to import queries after moving all the users over. The part with the following doesn't work correctly:
user_api_key = get_api_key(orig_client, query["user"]["id"])
user_client = Redash(DESTINATION, user_api_key)
response = user_client._post("/api/queries", json=data)
It gives a 400 error I think because technically that user doesn't exist yet. Is there any way around that or to face those users to be real users and not pending?
Hi @JSpenced, thanks for your efforts! This PR is still WIP so it definitely doesn't work in its current form. We'll post both here and on our user form (discuss.redash.io) once it's ready for testing (sometime in August, probably).
Force pushed changes from local dev.
Hi, first of all thanks for your efforts!
Regards
Missing extra step after migrate queries and before dashboards
redash-migrate visualizations
Missing extra step after migrate queries and before dashboards
redash-migrate visualizations
Fixed! Thanks :)
Hi, first of all thanks for your efforts!
- Will you add a feature for importing disabled users? It seems the queries that have been created by disabled users cannot be migrated.
- Please consider importing all the content (queries/users) in reversed created_at order. Because e.g. the latest created query/user in the origin instance appears in the last page of the destination instance.
Regards
I incorporated the chronological import for queries in 8a65452. Can you see how well this works for your needs?
Yes, I think we can import disabled users. Just requires a separate request to the UserListResource
. Should we disable them again in the new instance?
Yes, I think we can import disabled users. Just requires a separate request to the
UserListResource
. Should we disable them again in the new instance?
I think if the goal of the migration script is to get the destination instance as close as possible to the original state, then I would expect the disabled users to be migrated too.
At the moment, if the user is disabled, the queries that are owned by that user are not migrated either.
I incorporated the chronological import for queries in 8a65452. Can you see how well this works for your needs?
This is exactly what we need but in my opinion, implementing this feature to all migrated content (queries, users, dashboards etc.) will be much better..
This is exactly what we need but in my opinion, implementing this feature to all migrated content (queries, users, dashboards etc.) will be much better..
Done! See 623323d
I think if the goal of the migration script is to get the destination instance as close as possible to the original state, then I would expect the disabled users to be migrated too.
Agreed. Disabled user import (and a command to disable them again) is part of f8a6f6e
I'd like to merge these changes and put out documentation within the next week in advance of the V10 OSS Redash release. If you have more feedback best to provide it sooner than later 👍
Tried with the latest version. Here are some observations.
Query migration
Maybe the following needs to be False? Since we are importing ALL users, and then disabling them
https://github.com/getredash/redash-toolbelt/blob/issue-5/redash_toolbelt/examples/migrate.py#L155
Added a PR: https://github.com/getredash/redash-toolbelt/pull/53
Upon further digging the second error is caused by the following, not quite sure what this means.
{"message": "You are trying to associate a dropdown query that does not have a matching group. Please verify the dropdown query id you are trying to associate with this query."}
Favorites migration
Favorites - ERROR - list index out of range
I think this is caused by users having 0 migrations.
Query 720842 - FAIL - 400 Error - Destination user 14 is disabled. Query create failed!
It's unusual that a destination user is disabled before the query import is complete. Did you disable them manually? Or have you already run redash-migrate disable_users
? Either way, if you re-enable this user and try again the query should import.
Maybe the following needs to be False? Since we are importing ALL users, and then disabling them
https://github.com/getredash/redash-toolbelt/blob/issue-5/redash_toolbelt/examples/migrate.py#L155
Added a PR: #53
This line isn't the culprit. I commented on your PR.
- Query 840430 - FAIL - 400 Client Error: BAD REQUEST for url: http://localhost:8080/api/queries
Upon further digging the second error is caused by the following, not quite sure what this means.
{"message": "You are trying to associate a dropdown query that does not have a matching group. Please verify the dropdown query id you are trying to associate with this query."}
Check that the query owner has access to the right data sources. That error means that the dropdown parameter used by this query hits a data source to which the query's owner does not have access.
Favorites migration
Favorites - ERROR - list index out of range
I think this is caused by users having 0 migrations.
I agree with you. We should handle this more gracefully.
* Query 840430 - FAIL - 400 Client Error: BAD REQUEST for url: http://localhost:8080/api/queries
Upon further digging the second error is caused by the following, not quite sure what this means.
{"message": "You are trying to associate a dropdown query that does not have a matching group. Please verify the dropdown query id you are trying to associate with this query."}
I think this is caused by the parametrized queries, where parameters are dynamic and are generated by the query. I would guess that the query that is used to generate the parameters dropdown was not imported yet.
Also, upon running the redash-migrate queries
against the same instance that had a number of queries fail, and I am now
seeing the following error. Randomly some queries fail with the following:
Query 720842 - FAIL - 400 Error - Destination user 128 is disabled. Query create failed!
All users are in pending state, can we not make them active right away or "auto-accept" the invitations?
I would guess that the query that is used to generate the parameters dropdown was not imported yet.
I don't think so. That error is usually related to data source permissions. Have you configured any groups in your destination instance?
All users are in pending state, can we not make them active right away or "auto-accept" the invitations?
Pending users are considered active. Very odd that you are seeing disabled users unless you ran the disable_users
command or disabled them otherwise.
Just to confirm: which version of Redash are you migrating to?
Do you have any recommendations on exceeding 50 requests/hour limit of user API?
see related line
We are using one of the AMIs that redash provides here.
We have around 120 users. I think because of this limit, we cannot completely migrate our users. I am getting 429 too many requests error. Any recommendation rather than waiting 1 hour to migrate every 50 users? Can I disable the limit temporarily on AMI? Thank you!
I would guess that the query that is used to generate the parameters dropdown was not imported yet.
I don't think so. That error is usually related to data source permissions. Have you configured any groups in your destination instance?
Just double checked, we use just using the default users/admin groups. I manually added all data sources and all users to all the groups. However, You are trying to associate a dropdown query that does not have a matching group. Please verify the dropdown query id you are trying to associate with this query.
still persists.
Looking at the code, the error that is printed seems to be false positive, since it checks a 400 status and the meta "is_disabled".
All users are in pending state, can we not make them active right away or "auto-accept" the invitations?
Pending users are considered active. Very odd that you are seeing disabled users unless you ran the
disable_users
command or disabled them otherwise.Just to confirm: which version of Redash are you migrating to?
At the moment, I just cloned the master for redash. I assume this is version 10
Looking at the code, the error that is printed seems to be false positive, since it checks a 400 status and the meta "is_disabled".
Agreed it looks like line 240 of migrate.py
is suppressing the real error. There is still a 400 but we can't see what that error is. Fix incoming.
@bmtKIA6 I've just pushed 271df03 which removes the bad error message catch. Can you give this a try and report back?
Secondarily, I'm checking if I can reproduce the error you receive. If not, would you be interested in a 1-on-1 debugging session over slack/zoom/google-meet/discord?
Do you have any recommendations on exceeding 50 requests/hour limit of user API?
see related line
We are using one of the AMIs that redash provides here.
We have around 120 users. I think because of this limit, we cannot completely migrate our users. I am getting 429 too many requests error. Any recommendation rather than waiting 1 hour to migrate every 50 users? Can I disable the limit temporarily on AMI? Thank you!
Hi @ozgenbaris1 thanks for your question. We recommend you remove the rate-limit for the duration of the import process. To do this:
/opt/redash/env
.REDASH_RATELIMIT_ENABLED
. Set it to "false"
.docker-compose up -d server
."true"
using these steps.Secondarily, I'm checking if I can reproduce the error you receive. If not, would you be interested in a 1-on-1 debugging session over slack/zoom/google-meet/discord?
@bmtKIA6 I've just pushed 271df03 which removes the bad error message catch. Can you give this a try and report back?
Secondarily, I'm checking if I can reproduce the error you receive. If not, would you be interested in a 1-on-1 debugging session over slack/zoom/google-meet/discord?
I see no issues with that. Let me know what is the best time.
Hi there,
Thank you for the work.
Unfortunately, I have some difficulties to migrate my queries. 37 are OK, but 49 are KO with this message:
Query xxxxxx - FAIL - 400 Client Error: BAD REQUEST for url: https://redash.*************.com/api/queries
I've tried multiple things but no luck.
I've checked:
I've also tried to get more details about the errors, but one more time, no luck. DEBUG mode is enabled in Redash, but I don't see anything more than that in the container "redash_server_1":
Sample
... [2021-09-08 08:00:01,200][PID:11][DEBUG][metrics] table=organizations query=select duration=0.91 [2021-09-08 08:00:01,201][PID:11][DEBUG][root] Current organization: Elements (1) (slug: default) [2021-09-08 08:00:01,203][PID:11][DEBUG][metrics] table=users query=select duration=0.95 [2021-09-08 08:00:01,205][PID:11][DEBUG][metrics] table=groups query=select duration=0.67 [2021-09-08 08:00:01,207][PID:11][DEBUG][metrics] table=data_sources query=select duration=0.78 [2021-09-08 08:00:01,208][PID:11][DEBUG][metrics] table=data_source_groups query=select duration=0.57 [2021-09-08 08:00:01,210][PID:11][DEBUG][metrics] table=data_source_groups query=select duration=0.60 [2021-09-08 08:00:01,211][PID:11][DEBUG][metrics] table=groups query=select duration=0.61 [2021-09-08 08:00:01,212][PID:11][INFO][metrics] method=POST path=/api/queries endpoint=queries status=400 content_type=application/json content_length=177 duration=8.87 query_count=7 query_duration=5.09 [2021-09-08 08:00:01,229][PID:11][DEBUG][metrics] table=organizations query=select duration=0.96 [2021-09-08 08:00:01,230][PID:11][DEBUG][root] Current organization: Elements (1) (slug: default) [2021-09-08 08:00:01,232][PID:11][DEBUG][metrics] table=users query=select duration=1.47 [2021-09-08 08:00:01,235][PID:11][DEBUG][metrics] table=groups query=select duration=0.70 [2021-09-08 08:00:01,237][PID:11][DEBUG][metrics] table=data_sources query=select duration=0.83 [2021-09-08 08:00:01,238][PID:11][DEBUG][metrics] table=data_source_groups query=select duration=0.58 [2021-09-08 08:00:01,240][PID:11][DEBUG][metrics] table=data_source_groups query=select duration=0.53 [2021-09-08 08:00:01,241][PID:11][DEBUG][metrics] table=groups query=select duration=0.61 [2021-09-08 08:00:01,242][PID:11][INFO][metrics] method=POST path=/api/queries endpoint=queries status=400 content_type=application/json content_length=177 duration=9.12 query_count=7 query_duration=5.68 [2021-09-08 08:00:01,260][PID:11][DEBUG][metrics] table=organizations query=select duration=0.93 [2021-09-08 08:00:01,260][PID:11][DEBUG][root] Current organization: Elements (1) (slug: default) [2021-09-08 08:00:01,262][PID:11][DEBUG][metrics] table=users query=select duration=1.06 [2021-09-08 08:00:01,264][PID:11][DEBUG][metrics] table=groups query=select duration=0.69 [2021-09-08 08:00:01,266][PID:11][DEBUG][metrics] table=data_sources query=select duration=0.81 [2021-09-08 08:00:01,268][PID:11][DEBUG][metrics] table=data_source_groups query=select duration=0.98 [2021-09-08 08:00:01,269][PID:11][DEBUG][metrics] table=data_source_groups query=select duration=0.59 [2021-09-08 08:00:01,271][PID:11][DEBUG][metrics] table=groups query=select duration=0.60 [2021-09-08 08:00:01,272][PID:11][INFO][metrics] method=POST path=/api/queries endpoint=queries status=400 content_type=application/json content_length=177 duration=9.22 query_count=7 query_duration=5.66
How can I get more information about what happens?
Adding more logs in the script would be great in my humble opinion. :)
Thank you.
PS: I'm looking forward to get some hints!
Kind regards,
Adding more logs in the script would be great in my humble opinion. :)
Thank you for the detailed report @NicolasEsteves! Your request for better logs is well-heard. Sadly the root issue here is Redash, not the migration script. The application doesn't return many helpful messages for bad API requests 😦 Will add this to the list of items to address after V10 is released (very soon).
So more traditional debugging is required.
Looking at the queries that fail: does there seem to be a rhyme or reason to it? Do they all use dropdown parameters that point to other queries? What do they have in common, if anything?
I see no issues with that. Let me know what is the best time.
@bmtKIA6 and @NicolasEsteves if you are interested in debugging 1-on-1 please contact me through the Redash.io contact form at https://redash.io/contact/ and we can arrange a synchronous time block. I work during business hours in the pacific timezone but am flexible depending on your location. As this migration script is high importance to us I'm highly motivated to get it working for both of you 😃
This offer goes out to anyone in this thread. I'll delete this comment after the beta period concludes.
Adding more logs in the script would be great in my humble opinion. :)
Thank you for the detailed report @NicolasEsteves! Your request for better logs is well-heard. Sadly the root issue here is Redash, not the migration script. The application doesn't return many helpful messages for bad API requests 😦 Will add this to the list of items to address after V10 is released (very soon).
So more traditional debugging is required.
Looking at the queries that fail: does there seem to be a rhyme or reason to it? Do they all use dropdown parameters that point to other queries? What do they have in common, if anything?
Thank you for your answer.
Well noted for the logs, I understand.
For the queries, yes; there is a pattern. All queries that fail are all dependent on dropdown parameters from other queries. Does that make you think to something? Any idea to move forward?
Regarding the debugging session, yes that's a good idea! I'll contact you soon via the form (not really available this week). Thank you very much.
Kind regards,
For the queries, yes; there is a pattern. All queries that fail are all dependent on dropdown parameters from other queries. Does that make you think to something? Any idea to move forward?
@NicolasEsteves That's quite helpful information. Can you try something for me? For a single query that doesn't import, execute the queries that its dropdowns depend on. Then try the import again. Does it work?
For the queries, yes; there is a pattern. All queries that fail are all dependent on dropdown parameters from other queries. Does that make you think to something? Any idea to move forward?
@NicolasEsteves That's quite helpful information. Can you try something for me? For a single query that doesn't import, execute the queries that its dropdowns depend on. Then try the import again. Does it work?
@susodapop I've executed the query that fills the dropdown (that's the same dropdown for all queries that are KO), and then I've executed one query, but the issue remains; no change when I try to migrate the queries... :(
Any other idea or thing to test?
Thank you very much.
PS: We have checked all successfully migrated queries and I confirm, none of them have this dropdown, so are pretty sure this is the root cause.
Kind regards,
Looking at the queries that fail: does there seem to be a rhyme or reason to it? Do they all use dropdown parameters that point to other queries? What do they have in common, if anything?
Yes, all failing queries use other queries to populate the query parameters dropdowns.
Just wanted to confirm I received a couple emails this morning (PDT) about one-on-one debugging session and responded to them within the last few minutes.
Yes, all failing queries use other queries to populate the query parameters dropdowns.
and
I've executed the query that fills the dropdown (that's the same dropdown for all queries that are KO), and then I've executed one query, but the issue remains; no change when I try to migrate the queries... :(
This is excellent information, thank you both! I will play around with this and see if I can reproduce the error.
👋 I've been using the import tool pretty successfully.
Here is some feedback:
Importing queries with parameters using Query-based dropdowns seems to fail.
I've gotten this error when running redash-migrate queries
on some queries:
FAIL - 400 Client Error: BAD REQUEST for url:
After checking the said queries I realized all of them had parameters defined as Dropdowns based on query results. Some also contained emojis in them, which I thought could also be an issue, but I don't think that's the case.
What I tried to do to troubleshoot was to recreate manually the query at the destination with the same content and settings as in the source, and run redash-migrate queries
again. It now skips the query import fine, but the fixing now fails with a 403. I assume the latter error might be due to having created those queries by hand, but find that odd since I'm logged in with the same user as the one with the API key defined in the settings.
This led visualizations hence dashboards using those queries not to be fully imported.
Fortunately, we were not using too many of these so that won't be too much work to re-create, but it might be interesting to fix that.
Overall I'm happy to have found it and made it work, thanks for the script, that's been very helpful!
EDIT: Whoops, I just saw that mentioned above as I was late catching up on the thread!
After some 1-on-1 debugging with a customer I've pushed a provisional fix to the dropdown query import issues.
@thebiglabasky @bmtKIA6 and @NicolasEsteves Can you pull the latest commit for the issue-5
branch, rebuild with poetry, reinstall and try again? 🤞
Some of the queries fail to import due to a 400 error, now due to a token issue
Query 872110 - FAIL - 400 Client Error: BAD REQUEST for url: http://localhost:5000/api/queries
{'_content': b'{"message": "The CSRF token is missing."}', '
also, getting ONE 500 error, where queryId in options is None. Not quite sure what is so special about this query yet
Hey @bmtKIA6 on your 500 error do you see anything unusual about it on the origin instance?
I have an idea for how to fix the CSRF token missing error. We should never see this since CSRF tokens are not enforced for API calls. Which means the script is, for some reason, calling a non API endpoint...
Hey @bmtKIA6 on your 500 error do you see anything unusual about it on the origin instance?
I have an idea for how to fix the CSRF token missing error. We should never see this since CSRF tokens are not enforced for API calls. Which means the script is, for some reason, calling a non API endpoint...
For the ones that get 500, it looks like one of the queries that is populating the parameters is Archived. Are we migrating archived queries? From what I can tell there is a separate endpoint for the archived queries (/queries/archive
) which would lead me to believe that the archived queries are not migrated.
Archived queries are not being migrated, correct.
Archived queries are not being migrated, correct.
I think they should be, shouldn't they?
My understanding is that due to the lack of the "SQL dump", the goal of the migration script to have a carbon copy of the origin environment.
I was trying out this script today. On succesfully installing and setting meta.json file with values for origin_url
and destination_url
with secret keys. I was getting the below error.
Script failed during initialization.
Exception: invalid literal for int() with base 10: 'test'
Traceback (most recent call last):
File "/home/thebeauty/redash-toolbelt/env/bin/redash-migrate", line 8, in <module>
sys.exit(main())
File "/home/thebeauty/redash-toolbelt/env/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/thebeauty/redash-toolbelt/env/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/thebeauty/redash-toolbelt/env/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/thebeauty/redash-toolbelt/env/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/thebeauty/redash-toolbelt/env/lib/python3.8/site-packages/redash_toolbelt/examples/migrate.py", line 758, in main
from_client = Redash(ORIGIN, ORIGIN_API_KEY)
NameError: name 'ORIGIN' is not defined
@kurianbenoy-aot thanks for the feedback. To help debug further:
test
in your meta.json
file?My understanding is that due to the lack of the "SQL dump", the goal of the migration script to have a carbon copy of the origin environment.
@bmtKIA6 That's not the goal of the migration script. If it were, we would need to find a way to move query snippets, data source definitions, user groups, API keys, the whole lot...
The migration script is here to move active objects from one instance to another. Archived queries/dashboards are treated as deleted in the Redash front-end. They are only unarchived by an admin in the event of a mistaken delete. I think it does make sense to move them over -- and will need to think about how to accomplish this cleanly -- but lack of this feature shouldn't block release of toolbelt version of the migration script, which has never included this feature in the past.
@bmtKIA6 That's not the goal of the migration script. If it were, we would need to find a way to move query snippets, data source definitions, user groups, API keys, the whole lot...
The migration script is here to move active objects from one instance to another. Archived queries/dashboards are treated as deleted in the Redash front-end. They are only unarchived by an admin in the event of a mistaken delete. I think it does make sense to move them over -- and will need to think about how to accomplish this cleanly -- but lack of this feature shouldn't block release of toolbelt version of the migration script, which has never included this feature in the past.
ok. How about "Alert destinations"? The alerts are part of this migration, but not the destinations. Which results in non-functional alerts.
Should we recreate all of these manually?
After some 1-on-1 debugging with a customer I've pushed a provisional fix to the dropdown query import issues. @thebiglabasky @bmtKIA6 and @NicolasEsteves Can you pull the latest commit for the
issue-5
branch, rebuild with poetry, reinstall and try again? 🤞
Hi @susodapop, thank you very much for your work, I've been able to migrate the queries!
Few notes:
Now, I can see some differences with the Cloud version, especially for the display (eg. charts colors). Current version that I have is 8.0.0.b32245 but I can see this is not the latest (https://hub.docker.com/r/redash/redash/tags). May I know which version matches the most the Cloud one?
Thank you.
Kind regards,
Should we recreate all of these manually?
@bmtKIA6 Yes, alert destinations are moved the same way as data sources.
That said, the new alerts migration behaviour doesn't keep a mapping of old alert destinations and new ones. I think this will need to be introduced, otherwise the alerts will require manual fixes.
May I know which version matches the most the Cloud one?
@NicolasEsteves this script is designed to move from app.redash.io to Redash V10. We aren't testing it on V8.
May I know which version matches the most the Cloud one?
@NicolasEsteves this script is designed to move from app.redash.io to Redash V10. We aren't testing it on V8.
@susodapop I didn't know that, thank you. I've upgraded to V10. I think we are good for me from now on. Once again, thank you for the work, that helped a looooot!
Cheers
* some favorites couldn't be imported, but I didn't investigate as it's really not important to me.
- another question, Are tags for queries and dashboards intentionally left out? Talking to the users, this is the primary navigation for both queries and dashboards, and losing it in a migration is quite a pain
Not intended. Will research this.
I believe this fixes issue with favorites: https://github.com/getredash/redash-toolbelt/pull/57
It comes back to the fact that we are trying to import favorites for the disabled users, the request to /api/queries/favorites results in 404 and the client dies due to the use of raise_for_error()
I have troubles with user transfer from hosted instance to upensource with SAMl enabled.
I am receiving rhis error
Could not create user: probably this user already exists but is not present in meta.json importing: 367357 Could not create user: probably this user already exists but is not present in meta.json CAUTION: user list is not in sync! Destination contains 1 users. Origin contains 522 users Saving meta...
Usage
See DEVELOPING THIS SCRIPT in the readme.
Type of PR
Description
This effort adds a revised version of this gist that uses the
redash_toolbelt
client, supports recent versions of Redash and migrates all of the content (the gist was skipping alerts and favorites).Steps
2to3
client.py
pyproject.toml
Expanded scope
Related Tickets & Documents
5