Old dataset not included in dataset sheet won't go away

[longhotsummer]

We have an old dataset "contracts" which we don't want included in our census at http://za-city.census.okfn.org/. Our dataset google sheet at https://docs.google.com/spreadsheets/d/13CGUOt5eeqOfhYr3J52u34WQpIs8FiUlrCLgEqnw8TI/edit#gid=2 doesn't include "contracts" at all.

Refreshing the datasets via the admin page has no effect, the dataset won't go away.

[pwalsh]

@longhotsummer thanks. This is a bug, in the sense that he behaviour changed and it shouldn't have.

Still, we wouldn't expect datasets to be removed completely from a census: what about the history of submissions for a dataset - is it expected to be deleted if the dataset is removed from the sheet?

@morchickit / @vlevit fixing this bug is pretty easy, but requires additional tests. Currently, when loading data from spreadsheets, we perform upsert queries which add new records, and update existing records, but do not delete records that no longer exist in a sheet.

We need to refine this query strategy to handle cases as above, but first we do need a spec for it.

As a short term solution for @longhotsummer I'd suggest we simply remove the dataset record from the DB, and then next week we can write a spec for this issue and change the data loading strategy.

[longhotsummer]

As a short term solution for @longhotsummer I'd suggest we simply remove the dataset record from the DB, and then next week we can write a spec for this issue and change the data loading strategy.

This works for me. We haven't populated the census yet, we encountered this when testing.

Thanks!

[morchickit]

So, since it happen to me now as well in the Global census, this will need to be fixed ASAP (in my case, I change the name of the ID, which cause now to have double datasets name Geographical information). Let's fix it tomorrow. :-)

[pwalsh]

IDs absolutely cannot be changed - many things will break by changing an id - this was true in the old codebase and new codebase alike. NEVER change an ID for any object (place, question, dataset) that already has data.

@rgrp @vlevit

On 16 Aug 2015, at 16:44, morchickit notifications@github.com wrote:

So, since it happen to me now as well in the Global census, this will need to be fixed ASAP (in my case, I change the name of the ID, which cause now to have double datasets name Geographical information). Let's fix it tomorrow. :-)

— Reply to this email directly or view it on GitHub.

[morchickit]

Yes, totally my fault! I feel like an idiot @pwalsh - is this why the census now have an application error?

[pwalsh]

@vlevit I'm assigning to you. I need to right a spec for you for this. Let's sync on it in tomorrow's standup.

[longhotsummer]

Any progress here? It would be awesome if someone could just clean out our za-city census dataset and entry tables completely so we can start fresh, this bug is preventing us from user it.

[pwalsh]

@longhotsummer we are happy to get pull requests :).

It is scheduled for this week but not guaranteed: we accept it as a bug but in "normal" use it does not cause an issue.

Rest assured, we'll update here as soon as it is ready to ship. In the meantime, if you want me to clear out za-city datasets so you can reimport, I can.

BTW, this issue only covers datasets. If you've also created Entries with "bad" datasets, I can clear them out on this occasion too, but I would advise against using the census with dummy data :).

@morchickit you could possibly create some documentation pointing Census admins to the staging server, and we could encourage people to experiment there, and do something like create a script that clears and reloads the DB there daily. You'd need to open a new issue for this so we can assess the work.

[longhotsummer]

@pwalsh thanks for digging -- if you could clear out za-city that would unblock us, thanks.

I'll see if we can find time to send a pull request.

[pwalsh]

@longhotsummer I'm talking to @vlevit shortly and I'll be checking when we can start this. So, don't start any coding without first updating here and saying "hey, I want to look at this now".

[pwalsh]

@longhotsummer ok, we are working on this today. Also, I've just cleared out za-city datasets and entries.

[morchickit]

@pwalsh - sure, i can do it. Is it fine if I'll do it this afternoon?

[pwalsh]

@vlevit

Tasks

• [x] Ensure the current loader tests work
• [x] Refactor the loaders so that instead of upsert, we delete then insert for the given site.
• [x] Update tests for the new changes to the loaders
• [x] Extend test coverage if applicable (your call)
• [x] Make a PR and I'll check

[pwalsh]

FIXED.