apache / superset

Apache Superset is a Data Visualization and Data Exploration Platform
https://superset.apache.org/
Apache License 2.0
61.84k stars 13.54k forks source link

Import failed on duplicated key when importing existing chart that its dataset was changed #22506

Closed SharonCastel closed 8 months ago

SharonCastel commented 1 year ago

Every time we import a dashboard with chart that its dataset was changed, we're Getting the error: psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "uq_slices_uuid"

How to reproduce the bug

  1. export dashboard from source env and import it to target env
  2. On the source env:
    1. Go to dashboard
    2. Edit the chart
    3. change its dataset to another
    4. save (overwrite) the chart and the dashboard
    5. export the dashboard
  3. import the dashboard(2.5) on the target env with overwrite option.

Expected results

The import will overwrite the existing chart on the target env and update it to the new dataset (which by the way exists already on the target env)

Actual results

The import failed with the error: psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "uq_slices_uuid"

Screenshots

Traceback (most recent call last): File "/app/superset/cli/importexport.py", line 163, in import_dashboards ImportDashboardsCommand(contents, overwrite=True).run() File "/app/superset/dashboards/commands/importers/dispatcher.py", line 68, in run raise exc File "/app/superset/dashboards/commands/importers/dispatcher.py", line 57, in run command.run() File "/app/superset/commands/importers/v1/__init__.py", line 72, in run raise self.import_error() from ex superset.dashboards.commands.exceptions.DashboardImportError: Import dashboard failed for an unknown reason 2022-12-22 10:57:29,174:ERROR:superset.cli.importexport:There was an error when importing the dashboards(s), please check the exception traceback in the log Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context self.dialect.do_execute( File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute cursor.execute(statement, parameters) psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "uq_slices_uuid" DETAIL: Key (uuid)=(8eab9982-aaaa-450c-bbbb-1761d8c207f6) already exists.

Environment

Checklist

Make sure to follow these steps before submitting your issue - thank you!

Additional context

When I delete the chart from the target env and reimport it succeeded (unless there is another chart with the same change) In the code I see there is a select from tables before the insert into slices which probably makes that issue.

adimyth commented 1 year ago

Were you able to find a solution to this?

Zhichi57 commented 1 year ago

The problem is a query to the database that determines the existence of the chart. It contains the condition WHERE slices.datasource_id = %(datasource_id_1)s AND slices.uuid = %(uuid_1)s Because of this, the existence of the diagram is determined incorrectly and is not overwritten. The query execute in a file https://github.com/apache/superset/blob/master/superset/models/helpers.py#L300

SharonCastel commented 1 year ago

Yes you right, it can query by the uuid only I believe. Do you know if it fixed on version 2.0.2?

ShaliniIruvuru commented 1 year ago

It is working fine with owners permission. If Dashboard dont have Owners(None) then it is asking for create new chart. other wise it is working fine. Screenshots:- Screenshot from 2023-03-22 21-21-38 image

SharonCastel commented 1 year ago

Hi All, I checked it again with owners on the new version 2.1.0 and unfortunately the issue persists :( when the dataset on the import file is different than the one on the target chart the import of the dashboard failed from unknown reason: image This is critical feature in order to have life cycle for superset dashboards.

ShaliniIruvuru commented 1 year ago

Hi SharonCastel,

I have also checked it again with owners on the new version 2.1.0 .I didn’t get any error,it’s working fine.

Loom video:- https://www.loom.com/share/2e9f5d32fae44af592f0c0924b5905c5

SharonCastel commented 1 year ago

Hi @ShaliniIruvuru, The edit is working well, but the problem is when you change the data set to another. As I wrote before: when the dataset on the import file is different than the one on the target chart the import of the dashboard failed from unknown reason. So instead of "edit dataset" please check the "swap dataset".

ShaliniIruvuru commented 1 year ago

Hi @SharonCastel ,Checked in swap dataset also.I didn't get any error.Just i followed which you added that steps in how to reproduce the bug. Loom video:- https://www.loom.com/share/3c0d421d3dd5419e8ed437f6aeb107e6

SharonCastel commented 1 year ago

I watched the video but didnt understand which file did you import.. unfortunately if its the same file as you just exported so it wont reproduce the issue..

ShaliniIruvuru commented 1 year ago

Yes @SharonCastel I used for importing changed dataset file.

Can you please provide detailed information along with steps.It is more helpful if you provide video.

SharonCastel commented 1 year ago

How to reproduce the bug export the dashboard - save it to the last step Go to dashboard Edit the chart change its dataset to another - swap dataset save (overwrite) the chart and the dashboard import the file from the first step with overwrite option.

ShaliniIruvuru commented 1 year ago

Hi @SharonCastel ,I am not getting any error.I followed what ever you provided the steps and also providing the steps which I followed...Can you please go through 1.Export the dashboard(Chosen Sales Dashboard.Here selected one chart having cleaned sales data). image 2.Go to sales dashboard.

3.Edit the chart for Total revenue

4.Do Swap dataset (Here change the dataset to Birth names) image 5.Save (Overwrite) the chart and Dashboard.

6.Import the dashboard image

danikv commented 1 year ago

any updates on that issue ? I can confirm that it's still happening on the main branch.

SharonCastel commented 1 year ago

Export the dashboard(Cho

I believe you imported it on the same environment so the local id is the same and the issue wont bug you, but if you export one and import it to another environment it fails with duplicate error on the id which is the sequence of the db different on each env.

jfrag1 commented 1 year ago

https://github.com/apache/superset/pull/24821 was merged which should resolve this issue; it's currently tagged for the 2.1.2 and 3.0 releases, so it should be included in both of those.

SharonCastel commented 8 months ago

@jfrag1 you are the best!!! thank you so much for the fix!