Pull in new batch of data

[prabhasp]

This will require some co-ordination between @zmyao88 @schafik @myf @mejymejy @rowo I propose that we meet on Thursday 9am New York time / 2pm Abuja time / 9pm Indonesia time to figure this out.

Objectives:

• get the newest version of facility list in
• get IDs into the lists, as specified in #58

Background work to keep in mind:

• 29 - @zmyao88 added random_id and seq_id to csvs in dropbox

• 41 - @schafik changed headers

[prabhasp]

I updated Dropbox/Nigeria/Nigeria 661 Baseline Data Cleaning/in_process_data/*.csv; everything is now even more streamlined in terms of naming conventions than before. Also, put up a new branch https://github.com/modilabs/mopup-seed/tree/new_data_structure which deals with the new data structure in the front-end. But getting a "database error" that I might need @myf's help to resolve.

Outstanding:

• add id to the things from the nmis list. i'll do that soon and update this comment.
• fix the database error. @myf, need help on this one. will get in touch over irc.
• improve the load process. this one is ny team's challenge for today. mongoimport at present is dropping 12K of baseline hospitals csv, 55K (!) of baseline schools csv, as well as 4 and 12 of info from the facility lists on my install, which is not acceptable at all at all.

[prabhasp]

@myf tried to get in touch over irc, no luck. super tired, going to bed. ids are not yet resolved; will finish that tomorrow. meanwhile, salah is going to refresh the data once more. we should figure out csv -> mongo issues on new or old data both. the import script should make sure not to destroy data about matches that have been previously recorded. (note that we will continue to get facility lists data even as matching process has officially begun).

[myf]

will pull the new structure when you are sleeping.

On Thu, May 2, 2013 at 11:14 AM, prabhasp notifications@github.com wrote:

@myf https://github.com/myf tried to get in touch over irc, no luck. super tired, going to bed. ids are not yet resolved; will finish that tomorrow. meanwhile, salah is going to refresh the data once more. we should figure out csv -> mongo issues on new or old data both. the import script should make sure not to destroy data about matches that have been previously recorded. (note that we will continue to get facility lists data even as matching process has officially begun).

— Reply to this email directly or view it on GitHubhttps://github.com/modilabs/mopup-seed/issues/62#issuecomment-17343925 .

Have you ever imagined that you are standing on the EARTH?

[schafik]

data refreshed. @zmyao88 and I still have to sort out a couple of details, which we are in the process of doing. for now, feel free to use the .csvs there.

[myf]

@schafik @zmyao88 what are those random ids? roger's looking for sequential list for each lga for ui purposes.

On Thu, May 2, 2013 at 1:42 PM, schafik notifications@github.com wrote:

data refreshed. @zmyao88 https://github.com/zmyao88 and I still have to sort out a couple of details, which we are in the process of doing. for now, feel free to use the .csvs there.

— Reply to this email directly or view it on GitHubhttps://github.com/modilabs/mopup-seed/issues/62#issuecomment-17352865 .

Have you ever imagined that you are standing on the EARTH?

[myf]

problem with missing data: * quote inside in fields in baseline, getting rid of it then mongoimport works wonders implementing new data * data-import branch * actually pretty computational intensive, check if it exist, then insert, or not, for 80,000 records, but in the porcess

[myf]

@prabhasp @schafik @zmyao88 i am at my wit's end, the random id is not random nor unique. it was causing a lot of trouble for me with the assumption of it being the id. so if the random id is not random i am going to use mongo id and make that official long_id of the databsse. you guys have to import from me afterwards.