search

kinisi / geoserver

MIT License
0 stars 0 forks source link

CSV Downloader Exports Null/Empty Values #19

Open aeratek opened 9 years ago

aeratek commented 9 years ago

The CSV download functionality is exporting empty entries, which become difficult to work with in a spreadsheet (and in my CSV importer code I'm working with). Example shown at bottom.

The empty values come from one or more of the following: speed, altitude, climb, track, epx, epy, epv.

There's a few ways of approach a fix to this: (1) input the string "NULL" in place of the emptiness between commas, just like the GPS dongle does when it provides raw data in CSV format to the NodeJS app (I think), (2) add a new button or otherwise replace this button's functionality in the dashboard with the functionality of only exporting measure_time, latitude and longitude, for the time being, or (3) do not fix and make it the customer's problem (whoever downloads the CSV has to reformat it, removing these empty areas and aligning columns/rows manually/automatically somehow).

Mon Mar 16 10:00:36 EDT 2015,40.71669957,-73.965549512,1.23,34.83,-0.255,71.7706,15.02,22.024,89.408 Mon Mar 16 10:00:41 EDT 2015,40.71676894,-73.965521658,0.172,25.879,-0.398,274.1072,33.364,64.557,320.055 Mon Mar 16 10:00:46 EDT 2015,40.716646362,-73.965659989,4.261,,,219.897,33.364,64.557, Mon Mar 16 10:00:51 EDT 2015,40.716503709,-73.965838982,3.998,,,212.2336,,, Mon Mar 16 10:00:56 EDT 2015,40.716502086,-73.965827608,0.245,,,258.3136,,,

aohren commented 9 years ago

Sure, we can put placeholder values in for empty data if that's better. Most CSV data I've ever worked with uses this convention though (just an empty field) to represent missing values, instead of putting in a placeholder string like "NULL" in there.

FWIW, I imported this sample output in Google Spreadsheets and locally in OpenOffice and the data looks fine (appropriate fields are blank, just like the data).

On Fri, Apr 3, 2015 at 2:37 PM, aeratek notifications@github.com wrote:

The CSV download functionality is exporting empty entries, which become difficult to work with in a spreadsheet (and in my CSV importer code I'm working with). Example shown at bottom.

The empty values come from one or more of the following: speed, altitude, climb, track, epx, epy, epv.

There's a few ways of approach a fix to this: (1) input the string "NULL" in place of the emptiness between commas, just like the GPS dongle does when it provides raw data in CSV format to the NodeJS app (I think), (2) add a new button or otherwise replace this button's functionality in the dashboard with the functionality of only exporting measure_time, latitude and longitude, for the time being, or (3) do not fix and make it the customer's problem (whoever downloads the CSV has to reformat it, removing these empty areas and aligning columns/rows manually/automatically somehow).

Mon Mar 16 10:00:36 EDT 2015,40.71669957,-73.965549512,1.23,34.83,-0.255,71.7706,15.02,22.024,89.408 Mon Mar 16 10:00:41 EDT 2015,40.71676894,-73.965521658,0.172,25.879,-0.398,274.1072,33.364,64.557,320.055 Mon Mar 16 10:00:46 EDT 2015,40.716646362,-73.965659989,4.261,,,219.897,33.364,64.557, Mon Mar 16 10:00:51 EDT 2015,40.716503709,-73.965838982,3.998,,,212.2336,,, Mon Mar 16 10:00:56 EDT 2015,40.716502086,-73.965827608,0.245,,,258.3136,,,

— Reply to this email directly or view it on GitHub https://github.com/kinisi/geoserver/issues/19.

aohren commented 9 years ago

Also, what clients / apps are having problems with this exactly (ie- what CSV import lib / spreadsheet app are you using)? Might be worth figuring out what they expect for empty fields. For example it might just be Excel expecting quoting the fields (I think that's MS-style).

Inserting a placeholder might have it's own problems. Like if you try importing the CSV into a database, with the empty fields, that's fine. But if you have "NULL" or some other value in there, you'll have the word "NULL" all over your dataset and that'll require it's own cleaning/reformatting.

On Sat, Apr 4, 2015 at 9:54 AM, Adam Ohren adam.ohren@gmail.com wrote:

Sure, we can put placeholder values in for empty data if that's better. Most CSV data I've ever worked with uses this convention though (just an empty field) to represent missing values, instead of putting in a placeholder string like "NULL" in there.

FWIW, I imported this sample output in Google Spreadsheets and locally in OpenOffice and the data looks fine (appropriate fields are blank, just like the data).

On Fri, Apr 3, 2015 at 2:37 PM, aeratek notifications@github.com wrote:

The CSV download functionality is exporting empty entries, which become difficult to work with in a spreadsheet (and in my CSV importer code I'm working with). Example shown at bottom.

The empty values come from one or more of the following: speed, altitude, climb, track, epx, epy, epv.

There's a few ways of approach a fix to this: (1) input the string "NULL" in place of the emptiness between commas, just like the GPS dongle does when it provides raw data in CSV format to the NodeJS app (I think), (2) add a new button or otherwise replace this button's functionality in the dashboard with the functionality of only exporting measure_time, latitude and longitude, for the time being, or (3) do not fix and make it the customer's problem (whoever downloads the CSV has to reformat it, removing these empty areas and aligning columns/rows manually/automatically somehow).

Mon Mar 16 10:00:36 EDT 2015,40.71669957,-73.965549512,1.23,34.83,-0.255,71.7706,15.02,22.024,89.408 Mon Mar 16 10:00:41 EDT 2015,40.71676894,-73.965521658,0.172,25.879,-0.398,274.1072,33.364,64.557,320.055 Mon Mar 16 10:00:46 EDT 2015,40.716646362,-73.965659989,4.261,,,219.897,33.364,64.557, Mon Mar 16 10:00:51 EDT 2015,40.716503709,-73.965838982,3.998,,,212.2336,,, Mon Mar 16 10:00:56 EDT 2015,40.716502086,-73.965827608,0.245,,,258.3136,,,

— Reply to this email directly or view it on GitHub https://github.com/kinisi/geoserver/issues/19.

aeratek commented 9 years ago

Gotcha. Ok, let's hold off on implementing this, then, given what you mentioned. I will look at what happens when importing these CSV and KML files into tools like CartoDB and Google Earth, when they have empty/null values, just to double-check there's no issues. Both tools are usually pretty good about this. In my case, I was attempting to import this data into my own (Java) app and I really only needed measure_time, lat and lon, so the other data wasn't doing much for me and was just causing CSV-parsing complexity for the simple mapping and analysis I was doing.

On Sat, Apr 4, 2015 at 3:54 AM, Adam Ohren notifications@github.com wrote:

Sure, we can put placeholder values in for empty data if that's better. Most CSV data I've ever worked with uses this convention though (just an empty field) to represent missing values, instead of putting in a placeholder string like "NULL" in there.

FWIW, I imported this sample output in Google Spreadsheets and locally in OpenOffice and the data looks fine (appropriate fields are blank, just like the data).

On Fri, Apr 3, 2015 at 2:37 PM, aeratek notifications@github.com wrote:

The CSV download functionality is exporting empty entries, which become difficult to work with in a spreadsheet (and in my CSV importer code I'm working with). Example shown at bottom.

The empty values come from one or more of the following: speed, altitude, climb, track, epx, epy, epv.

There's a few ways of approach a fix to this: (1) input the string "NULL" in place of the emptiness between commas, just like the GPS dongle does when it provides raw data in CSV format to the NodeJS app (I think), (2) add a new button or otherwise replace this button's functionality in the dashboard with the functionality of only exporting measure_time, latitude and longitude, for the time being, or (3) do not fix and make it the customer's problem (whoever downloads the CSV has to reformat it, removing these empty areas and aligning columns/rows manually/automatically somehow).

Mon Mar 16 10:00:36 EDT

2015,40.71669957,-73.965549512,1.23,34.83,-0.255,71.7706,15.02,22.024,89.408 Mon Mar 16 10:00:41 EDT

2015,40.71676894,-73.965521658,0.172,25.879,-0.398,274.1072,33.364,64.557,320.055 Mon Mar 16 10:00:46 EDT 2015,40.716646362,-73.965659989,4.261,,,219.897,33.364,64.557, Mon Mar 16 10:00:51 EDT 2015,40.716503709,-73.965838982,3.998,,,212.2336,,, Mon Mar 16 10:00:56 EDT 2015,40.716502086,-73.965827608,0.245,,,258.3136,,,

— Reply to this email directly or view it on GitHub https://github.com/kinisi/geoserver/issues/19.

— Reply to this email directly or view it on GitHub https://github.com/kinisi/geoserver/issues/19#issuecomment-89520576.

aohren commented 9 years ago

Cool. Another thing we can do is have a request specify which fields they want. So like if you only want time, lat, and long, then it could exclude all the other fields in the download.

On Saturday, April 4, 2015, aeratek notifications@github.com wrote:

Gotcha. Ok, let's hold off on implementing this, then, given what you mentioned. I will look at what happens when importing these CSV and KML files into tools like CartoDB and Google Earth, when they have empty/null values, just to double-check there's no issues. Both tools are usually pretty good about this. In my case, I was attempting to import this data into my own (Java) app and I really only needed measure_time, lat and lon, so the other data wasn't doing much for me and was just causing CSV-parsing complexity for the simple mapping and analysis I was doing.

On Sat, Apr 4, 2015 at 3:54 AM, Adam Ohren <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Sure, we can put placeholder values in for empty data if that's better. Most CSV data I've ever worked with uses this convention though (just an empty field) to represent missing values, instead of putting in a placeholder string like "NULL" in there.

FWIW, I imported this sample output in Google Spreadsheets and locally in OpenOffice and the data looks fine (appropriate fields are blank, just like the data).

On Fri, Apr 3, 2015 at 2:37 PM, aeratek <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

The CSV download functionality is exporting empty entries, which become difficult to work with in a spreadsheet (and in my CSV importer code I'm working with). Example shown at bottom.

The empty values come from one or more of the following: speed, altitude, climb, track, epx, epy, epv.

There's a few ways of approach a fix to this: (1) input the string "NULL" in place of the emptiness between commas, just like the GPS dongle does when it provides raw data in CSV format to the NodeJS app (I think), (2) add a new button or otherwise replace this button's functionality in the dashboard with the functionality of only exporting measure_time, latitude and longitude, for the time being, or (3) do not fix and make it the customer's problem (whoever downloads the CSV has to reformat it, removing these empty areas and aligning columns/rows manually/automatically somehow).

Mon Mar 16 10:00:36 EDT

2015,40.71669957,-73.965549512,1.23,34.83,-0.255,71.7706,15.02,22.024,89.408

Mon Mar 16 10:00:41 EDT

2015,40.71676894,-73.965521658,0.172,25.879,-0.398,274.1072,33.364,64.557,320.055

Mon Mar 16 10:00:46 EDT 2015,40.716646362,-73.965659989,4.261,,,219.897,33.364,64.557, Mon Mar 16 10:00:51 EDT 2015,40.716503709,-73.965838982,3.998,,,212.2336,,, Mon Mar 16 10:00:56 EDT 2015,40.716502086,-73.965827608,0.245,,,258.3136,,,

— Reply to this email directly or view it on GitHub https://github.com/kinisi/geoserver/issues/19.

— Reply to this email directly or view it on GitHub https://github.com/kinisi/geoserver/issues/19#issuecomment-89520576.

— Reply to this email directly or view it on GitHub https://github.com/kinisi/geoserver/issues/19#issuecomment-89581365.

from mobile; forgive brevity