Total datasets number on dashboard page not correct

[anuveyatsu]

The number of datasets should increase when a new dataset is pushed, however, if it fails then it is not counted.

How to reproduce

• push some invalid data
• wait until it fails
• see no difference on datasets number on the dashboard page

Expected behaviour

• total number of datasets increased even if a newly pushed dataset fails

[zelima]

@anuveyatsu this raises another question for me - What should we display on the dashboard page and what on the publisher's page? Should those numbers differ? Saying cause if we change current approach, they will.

Currently, dashboard gets the total number of datasets from search API (so does publisher page), cause we simply do not serve any other endpoint to get that number. So if we change this and get it Eg from specstore those numbers will be different as in search only successful datasets appear.

From here comes another question - to see all datasets currently user clicks the username and is redirected to publisher page. Where should the user be redirected to see all datasets (failed and successful ones together) in this case? The publisher page cannot get a list of the dataset from elsewhere then search API, right?

[zelima]

One option would be to export failed datasets as well to metastore and add new flag success=true/false, so that frontend is able to filter. but this will require changes in metastore, remapping and reindexing of elasticsearch - Is it worth fixing?

[anuveyatsu]

@zelima Yes, I think all Failed datasets should be considered private and should be treated as regular private dataset. This way the only owner can see it on the list.

[zelima]

@AcckiyGerman This should be fixed. We now export metadata even if it's failed with unlisted findability (Only if there's no successful revision)

[AcckiyGerman]

TESTED & FIXED: Datasets with errors are included in Total datasets number on Publisher's page.