Import Work Orders Feature

[aarontitus]

Original author: v...@aarontitus.net (January 13, 2013 21:56:21)

There needs to be a method to import hundreds/ thousands of records using a combination of a standard, downloadable .csv or .xls/.xlsx file (which updates as the database schema is updated) and wizard to fix records with errors in addresses, Claimed organization, status, etc. The feature must be very fault tolerant if it is going to be available to all participants.

Alternatively, a less fault-tolerant import feature could be implemented if available ONLY to the administrator. The system would rely upon the administrator to fix any logical errors in the combination of status and claim.

This is a very high priority.

Original issue: http://code.google.com/p/sandy-disaster-recovery/issues/detail?id=133

[aarontitus]

From v...@aarontitus.net on March 11, 2013 15:27:52 Yes. it would be better to have the error message column prepended to the start of each row.

[aarontitus]

From cpw...@gmail.com on March 12, 2013 13:15:25 All the above should now be fixed (r228, v138), with the exception of unpredictable geocoding errors.

I have tested using the concatenated Rockaways data on the testbed (842 rows total)

[aarontitus]

From cpw...@gmail.com on March 12, 2013 15:33:06 I am yet to encounter a geocoder quota error whilst testing.

So, rather than add custom logic, there is now a rerun method (r230, v139, live on testbed) to allow the admin to manually rerun a CSV file if the geocoder fails in one 24 hours period.

Additionally, this allows the admin to add missing organizations and rerun without deleting and re-uploading a CSV file.

[aarontitus]

From cpw...@gmail.com on March 12, 2013 15:38:24 Of course, we can add automatically handling of geocoder outages and overrages if it becomes a big problem, later.

[aarontitus]

From v...@aarontitus.net on March 14, 2013 01:19:57 BUG: On sandbox, Test Incident, search for Work Order B1797. You'll notice that if you click on the icon, no InfoBox appears.

I looked at the original data, and this person does not have a telephone number (a required item). So it looks like the system imports invalid rows AND marks them as invalid. Then we get a that strange error when a required field is not present. I confirmed this behavior with B1805, who lacked a phone number. I also confirmed this behavior for B2161, which lacked a valid address.

After we fix this bug, can we push these fixes live so I can import the Rockaways to the live site?

[aarontitus]

From cpw...@gmail.com on March 14, 2013 11:18:34 I need extra details here - what page are you clicking an icon on?

(a) On /edit?id= , infoboxes appear in no cases. Should they? (b) On /map there are lots of DOM errors - but with no immediately obvious pattern.

It's not clear that this is related to imports - testing more now.

FYI there's no notion of "marking invalid" - CSV rows are either imported as site objects / work orders or not.

CC Andy because this has become about maps.

[aarontitus]

From cpw...@gmail.com on March 14, 2013 11:44:40 I am assuming it's (b) - added as #217 because I don't think it's related to this.

[aarontitus]

From cpw...@gmail.com on March 14, 2013 13:26:36

217 fixed - was that the problem?

[aarontitus]

From cpw...@gmail.com on March 14, 2013 13:35:48 Re #56 and #57 here: to be clear, I ignored the suggestion of the possible cause because on the test incident on the testbed,

B1797 has a phone number B1805 has a phone number B2161 has an address

[aarontitus]

From v...@aarontitus.net on March 14, 2013 14:03:31 So the way it worked- When I first uploaded B1797, B1805, and B2161, they did NOT have a given piece of information, and caused an error on import. I added the information and uploaded a second time.

Unfortunately I don't have time to test all of the various possibilities at the moment, so I'm OK with pushing it live now that you've fixed #217. I'd like you to go live with the current release because there are 400-500 records from Mississippi and Georgia that will need to be imported in the next few days.

[aarontitus]

From cpw...@gmail.com on March 14, 2013 14:09:14 As you added the information before the second import, there should have been no effect.

I will make the latest version live now.

[aarontitus]

From v...@aarontitus.net on March 14, 2013 14:43:13 Thanks!