Closed BasilHorowt closed 3 years ago
yeah we should just drop the colons in filenames.
colons were used as path separators in classic macos. i believe macs still treat them specially. we should just avoid them.
not sure about finer granularity than seconds. even when i'm running plotman tests using k25's i don't start plots more than once a second, and in production the point of plotman is largely to stagger jobs...
On Wed, Apr 21, 2021 at 9:24 AM BasilHorowt @.***> wrote:
What I'm seeing on OSX is that the plot filenames in the file browser (Finder.app) look like
2021-04-21-09/08/55.log
this is quite odd given the filesystem uses forward slashes as directories. When I check the dir in terminal via ls etc it looks like this:
2021-04-20-23\:13\:57.log
So I think perhaps because of my filesystem (APFS in this case) or the OS I'm seeing two totally different things which makes understanding and debugging and support more challenging.
Checking the source https://github.com/ericaltendorf/plotman/blob/c0fe10cf09e6b1d9ae04101e7babc1229d0c740f/src/plotman/manager.py#L105, i see it is using colons, so i know that is what plotman is saving.
The backslashes in CLI make sense (it does the same thing for spaces), however the display in Finder is really weird to me.
Thoughts on avoiding colons to make things more compatible? I think YYYYMMDD-HHMMSS would still sort nicely and be readable if needed. BTW should log names be more precise than seconds to avoid collisions ever?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ericaltendorf/plotman/issues/144, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARPZIFUBE2DRCOOKFFJZSRDTJ332NANCNFSM43KWBV4A .
I think Windows bans colons outright. It would be nice to use a standardized format, but I'm not sure one exists that is file name compatible. https://github.com/ericaltendorf/plotman/issues/108#issuecomment-817763506 ISO8601 with :
-> -
might be as close as we get. I'd personally keep the seconds. I'd rather have 60x the resolution I need than 0.9x. Maybe even the time zone. And yes, I did actually have two plotters set with different time zones... Regardless, I suppose the code should have a fall back in case of collision.
Oh definitely not suggesting dropping seconds, if anything being more precise. Timezone is an interesting consideration, would it make sense at that point to normalize them to UTC so they sort as you expect they would? I feel like the average user won't want to have to think about UTC, but perhaps they don't need to understand the log name convention anyways ?
I was figuring an option... :]
Kind of off-topic but... but on the 'is 1 second resolution good enough to be unique' topic... Note the duplicate entry supposedly for plot id ccd95602
with PIDs 301755 and 301756. Both those processes have log file 2021-04-25-11:20:41.log
open. 301756 does in fact have tmp files open for ccd95602
but 301755 is instead working on 857b4af7
.
They also commented that there had been a rogue plotman process running at some point which could explain same timestamp on two plotters. None the less...
@altendky and I talked about this for a bit and ended up with the plan to:
1) Replace :
with _
to ensure cross platform compatibility
2) Increase precision by adding microseconds to the end of the timestamp .123412
. This will ensure that collisions do not occur even at scale (someone could be plotting on many machines and consolidating logs for analysis?)
3) Appending the timezone to the end of the timestamp with +hh:mm
or -hh:mm
so that the log names have comprehensive time information present. We looked into it and +
shouldn't be a problem in the filename
What I'm seeing on OSX is that the plot filenames in the file browser (Finder.app) look like
2021-04-21-09/08/55.log
this is quite odd given the filesystem uses forward slashes as directories. When I check the dir in terminal via
ls
etc it looks like this:2021-04-20-23\:13\:57.log
So I think perhaps because of my filesystem (APFS in this case) or the OS I'm seeing two totally different things which makes understanding and debugging and support more challenging.
Checking the source, i see it is using colons, so i know that is what plotman is saving.
The backslashes in CLI make sense (it does the same thing for spaces), however the display in Finder is really weird to me.
Thoughts on avoiding colons to make things more compatible? I think YYYYMMDD-HHMMSS would still sort nicely and be readable if needed. BTW should log names be more precise than seconds to avoid collisions ever?