search

ut-data-maps / geoquery

1 stars 1 forks source link

Data arrangement #10

Open metasean opened 9 years ago

metasean commented 9 years ago

Based on the conversation with Wil tonight, I've written up a basic description of the data nodes that I'll be setting up within gun - https://github.com/ut-data-maps/geoquery/wiki/Node-Data

@wilbown - We talked specifically about three tables, one for the food, one for land parcels, and one for population. Is it easier for you to have those three distinct tables, or can you use the 'Type' value to separate out each node into its appropriate table?

@matwhite - I won't have a chance to work on this again until Monday. If you're anxious to get started, you can probably figure out how to start massaging data based on app.js lines 20 & L25, data.js lines 19 & L23, grocery.js lines 10-17, and the data.json file (you'll have to start up a local instance and open http://localhost:4200/ to generate it)

matwhite commented 9 years ago

Hi Sean,

Thanks for the head start on this. I am completely new to Gun, so it took me a while just to figure out that this uses nodejs and how to launch an instance. I'm not strong on JavaScript, so I have a dumb question... I am having trouble with the scoping when trying to get it to save data. Here is my code... the "tsv" var is still "empty" at the end (I want to save the data for writing to a file later).

var Gun = require('gun'); var gun = Gun({ file: 'data.json', s3: { key: '', // AWS Access Key secret: '', // AWS Secret Token bucket: '' // The bucket you want to save into } });

var fs = require('fs');

// Get data using same function as mapping app (code re-use) eval(fs.readFileSync('./app/scripts/data.js')+''); var socrata = gun.get('socrata').not(getSocrata);

// Loop through the "rows" of data to build CSV structure var tsv = 'empty'; // store TSV data socrata.map().val(function (s) { if (!s.location_1) { return; } var items = new Array(); items.push(s.name, s.naics, s.phone, s.ownership, s.emprangecode) var tsv_build = items.join("\t") + "\n"; tsv += tsv_build process.stdout.write(tsv_build); });

console.log(tsv);

Thanks.

--Mat

On 07/24/2015 02:23 AM, MetaSean wrote:

Based on the conversation with Wil tonight, I've written up a basic description of the data nodes that I'll be setting up within gun - https://github.com/ut-data-maps/geoquery/wiki/Node-Data

@wilbown https://github.com/wilbown - We talked specifically about three tables, one for the food, one for land parcels, and one for population. Is it easier for you to have those three distinct tables, or can you use the 'Type' value to separate out each node into its appropriate table?

@matwhite https://github.com/matwhite - I won't have a chance to work on this again until Monday. If you're anxious to get started, you can probably figure out how to start massaging data based on app.js lines 20 & L25 https://github.com/ut-data-maps/geoquery/blob/dev/app/scripts/app.js#L20-L25, data.js lines 19 & L23 https://github.com/ut-data-maps/geoquery/blob/dev/app/scripts/data.js#L19-L23, grocery.js lines 10-17 https://github.com/ut-data-maps/geoquery/blob/dev/app/scripts/grocery.js#L10-L17, and the |data.json| file (you'll have to start up a local instance and open http://localhost:4200/ to generate it)

— Reply to this email directly or view it on GitHub https://github.com/ut-data-maps/geoquery/issues/10.

metasean commented 9 years ago

@matwhite - :disappointed: I'm sorry I didn't explicitly point out the README.md (it would have saved you the whole node, npm, and spin-up challenges).

You were actually rockin' gun and JavaScript scope! You had a minor translation error (process.stdout.write should be something like console.log) and a more challenging JavaScript timing issue (the console.log(tsv); command was getting called before all the relevant calls had returned). I was having connectivity issues as I was addressing the latter and realized that the code wasn't as robust as it should be, so I cleaned that up.

You may have also run into a missing data problem. I hadn't seen it before, but ran into it several times tonight. So I also put in a little patch for that. A more robust solution needs to be implemented, but this patch should be good for the moment).

The working code is on the dev branch in commit 10c0f29c19622729358d3bc3a3be137d01627399.

matwhite commented 9 years ago

Hi Sean,

Thank you for taking a look at this and pointing me in the right direction :)

I used process.stdout.write in the loop for a couple of reasons: 1) it doesn't mess with the arg (adding a "\n" like console.log does), and 2) I wanted to watch the tsv var grow within the loop. The thing that keeps breaking my brain is that I can watch the tsv var grow inside the loop, but then when I call console.log(tsv) after the loop, tsv is still "empty." Yet according to scoping rules I feel like I had stored the data inside a global var. One other thing that is odd is that the output of the final console.log(tsv) seems to occur before the loop over the data, even though that last call is at the end of the code. So, I think there is some magic closure stuff going on in socrata.map().val() that I don't have a grasp on.

--Mat

On 08/01/2015 02:21 AM, MetaSean wrote:

@matwhite https://github.com/matwhite - :disappointed: I'm sorry I didn't explicitly point out the README.md https://github.com/ut-data-maps/geoquery/blob/dev/README.md (it would have saved you the whole node, npm, and spin-up challenges).

You were actually rockin' gun and JavaScript scope! You had a minor translation error https://github.com/ut-data-maps/geoquery/blob/10c0f29c19622729358d3bc3a3be137d01627399/app/scripts/app.js#L36-L39 (|process.stdout.write| should be something like |console.log|) and a more challenging JavaScript timing issue (the |console.log(tsv);| command was getting called before all the relevant calls had returned). I was having connectivity issues as I was addressing the latter and realized that the code wasn't as robust as it should be, so I cleaned that up https://github.com/ut-data-maps/geoquery/blob/10c0f29c19622729358d3bc3a3be137d01627399/app/scripts/app.js#L48-L56.

You may have also run into a missing data problem. I hadn't seen it before, but ran into it several times tonight. So I also put in a little patch https://github.com/ut-data-maps/geoquery/blob/10c0f29c19622729358d3bc3a3be137d01627399/app/scripts/grocery.js#L14-L16 for that. A more robust solution needs to be implemented, but this patch should be good for the moment).

The working code is on the dev branch in commit 10c0f29 https://github.com/ut-data-maps/geoquery/commit/10c0f29c19622729358d3bc3a3be137d01627399.

— Reply to this email directly or view it on GitHub https://github.com/ut-data-maps/geoquery/issues/10#issuecomment-126883931.