shark-parser file names

[d9i]

While working on the new spec for deataset folder structures, I realized the names shark-parser gives its output files could cause some problems down the line. I'll list these issues, then suggest an improved naming scheme. Let me know what you think/if we can change shark-parser to output names in this way.

example of a current filename: 2017-8-22 8/29/50_21a _data.csv

Issues:

• Currently, times are separated either by a colon or a forward slash. Forward slashes in filenames are a really big problem, because some parsers interpret them as a subdirectory, and Google Drive sometimes rewrites the file names to avoid them.
• Dates and times don't contain leading zeroes, which isn't compatible with ISO 8601. This isn't actually a problem for data analysis work, but it's not best practice, either.
• Dataset folder names use YYYYMMDD, while csv file names use YYYY-M-D, which is inconsistent
• Finally, it'd personally make me happier if data file names didn't include spaces, since I'd have to include fewer quotation marks in my shell scripts ;p (this might actually be impossible though, since tag names sometimes contain spaces)

In light of these, here's a suggestion for an improved filename: 2017-08-22_08-29-50_21a _data.csv

Alternatively (shorter and consistent with folder names): 20170822_082950_21a _data.csv

Would this be possible? Please let me know what you think!

[wdconinc]

ISO 8601 solves the spaces issue too. Even includes timezone.

$ date --iso-8601=seconds
2017-10-18T16:46:02-04:00                    

[TheBen27]

Prefer your first alternative, can't think of incompatibilities. Not a fan of the second, mainly because I find the folder dates difficult to read at a glance already.

If you're going to change the folder names tell me so we can have consistently naming.

Thanks for bringing this up.

[d9i]

Cool!

Here's some more confusion: William and I looked into adhering to ISO 8601 more closely. We can't actually name the file like 2017-08-22T08:29:50-05:00_21a _data.csv because colons aren't allowed in windows file names. The only other acceptable alternative in ISO 8601 is 20170822T0829500500_21a _data.csv, but that's very unreadable.

We can relax our adherence to 8601 and do 2017-08-22T08-29-50_21a _data.csv or 2017-08-22_08-29-50_21a _data.csv (which is what we seem to have agreed on so far), but that eschews time zone data.

If we want to include time zone data, there are a few questions: would we have to define that in the configure sketch? Would that show up in the header? Adding awareness of time zones seems somewhat non-trivial.

[d9i]

Also, once we figure out good file names, I'll change folder names to match. :)

[wdconinc]

I don't feel strongly. Don't mistake my commenting on this as an indication that I'm pushing a solution :-)

[d9i]

gotcha! :)

In that case: with your go-ahead, Ben, I'll add YYYY-MM-DD_hh-mm-ss_tagname_data.csv to the spec, change the data folder names, and then we can have the parser output dates in that format as well.

[cbilenkin]

Resolved by commit 3bd426d