Timestamp filename issue

[Joe-Heffer-Shef]

Putting the time in a filename can cause problems on different operating systems e.g. '%H:%M:%S' with colons will fail on Windows, which will affect a lot of potential research users.

Can we use hyphens or something else? e.g. decide a global two-way algorithm to convert a timestamp to a filename to be used across the whole project (all repos).

For example, a semicolon : ↔ ;

import datetime
from pathlib import Path

timestamp = datetime.datetime.now().isoformat()
filename = timestamp.replace(":", "+")
path = Path(f"photo_object_{filename}.np")

this gives photo_object_2024-03-18T14+49+56.465855.np and can easily be converted back to an ISO standard timestamp.

Note that this doesn't include the time zone, which means the assumption is that timestamps are in UTC.

[lionfish0]

• This has already caused problems for multiple students, so definitely a good idea.
• If you think semicolons are the best symbol then let's go with that. Thanks for sorting this! Feel free to make the code change.

Relevant code points:

• camera.py, line 201
• trigger.py, line 85
• there's also an attempt at sorting out zipping files (I think horrifyingly there's two separate implementations in the code for this functionality), that has relevance: file_manager.py line 21

[Joe-Heffer-Shef]

On reflection, a semicolon could cause headaches

A + will work better

To rename files

# Replace colon in filenames
for filename in *:*.np; do
  new_name=${filename//:/+}
  mv "$filename" "$new_name"
done

[lionfish0]

The filename/structure we maybe decided on (in the readme) says to scrap the separator completely!

<session>/<set>/<device id>/<camera id>/<timestamp>_<photo id>.np

• the 'session' is what calibration will work on -- it will assume all the photos from within the same session have the same alignment. If a camera gets moved, a new session folder will be needed.
• the 'set' is for the convenience of the research -- it could be e.g. "learningflight12" or "tomato23". A special set called 'cal' is for the use of the calibration tool, and contains photos of the calibration QR code for 3d camera registration.
• the 'device id' is a unique label for each of the "boxes".
• the 'camera id' is the id of the camera inside the box (there are sometimes two or more internal cameras).
• the 'timestamp' is a YYYMMDD_HHMMSS.UUUUUU string of when the photo was taken in the UTC time zone

[lionfish0]

• We've just replaced the code with the working original version from lionfish0's repo.
• This now will still be an issue.
• Joe/Euan when modifying the code for the new folders/filenames -- could you fix the time stamp syntax so it doesn't have : in any more. See last comment.

[jlwoodgate]

Could the '.' between the seconds and microseconds also cause issues? Maybe:

YYYYMMDD_HHMMSSUUUUUU.np

?

[Joe-Heffer-Shef]

Could the '.' between the seconds and microseconds also cause issues? Maybe:

YYYYMMDD_HHMMSSUUUUUU.np

No that dot is fine and won't break anything, and Python's pathlib module will handle it fine (see code below). In general, I'd recommend sticking as close as possible to an ISO standard timestamp which means that datetime.fromisoformat can be used.

>>> import datetime
>>> from pathlib import Path
>>>
>>> timestamp = datetime.datetime.now().isoformat()
>>> filename = timestamp.replace(":", "+")
>>> path = Path(f"photo_{filename}.np")
>>> path
WindowsPath('photo_2024-06-11T16+19+02.501853.np')
>>> path.suffix
'.np'
>>> path.stem
'photo_2024-06-11T16+19+02.501853'
>>> path.name
'photo_2024-06-11T16+19+02.501853.np'

and to parse the timestamp:

>>> _, _, timestamp = path.stem.partition('photo_object_')
>>> timestamp
'2024-06-11T16+19+02.501853'
>>> datetime.datetime.fromisoformat(timestamp.replace('+', ':'))
datetime.datetime(2024, 6, 11, 16, 19, 2, 501853)

[lionfish0]

Thanks!! Euan/JoeW do you want to look at this. Also - I just merged straight to main (Sorry) the code for timestamps from cameras. also removed junk from camera.py etc. https://github.com/SheffieldMLtracking/bee_track/issues/9#issuecomment-2161100571

On Tue, 11 Jun 2024 at 16:21, Joe Heffer @.***> wrote:

Could the '.' between the seconds and microseconds also cause issues? Maybe:

YYYYMMDD_HHMMSSUUUUUU.np

No that dot is fine and won't break anything, and Python's pathlib module will handle it fine (see code below). In general, I'd recommend sticking as close as possible to an ISO standard timestamp which means that datetime.fromisoformat https://docs.python.org/3/library/datetime.html#datetime.date.fromisoformat can be used.

import datetime>>> from pathlib import Path>>>>>> timestamp = datetime.datetime.now().isoformat()>>> filename = timestamp.replace(":", "+")>>> path = Path(f"photo_{filename}.np")>>> pathWindowsPath('photo_2024-06-11T16+19+02.501853.np')>>> path.suffix'.np'>>> path.stem'photo_2024-06-11T16+19+02.501853'>>> path.name'photo_2024-06-11T16+19+02.501853.np'

— Reply to this email directly, view it on GitHub https://github.com/SheffieldMLtracking/BBSRC_ohio/issues/12#issuecomment-2161035889, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4MGQC23R73AXVQIIBURULZG4IZDAVCNFSM6AAAAABEVWZ6TOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRRGAZTKOBYHE . You are receiving this because you commented.Message ID: @.***>

--

---

Mike Smith, Lecturer in Probabilistic Machine Learning

Working part time (usually 8am-2pm, Monday-Thursday). Department of Computer Science University of Sheffield

---