We are working to support multiple different datasets, however, as it stands indexing multiple datasets places them all in the same set of binary files. Ideally, we'd want to be able to subsample the data so we can combine different dataset indexes or split them apart trivially.
The solution would be to separate each dataset's observations into their own subdirectory and then separate each binary file by calendar month. This way downsampling the full dataset becomes a simple table query.
I've modified a code snippet from Stack Overflow that can be used to divide into calendar months (this could be a good starting place for any implementation).
# Modified from: https://stackoverflow.com/questions/51293632/how-do-i-divide-a-date-range-into-months-in-python
import datetime
import numpy as np
from astropy.time import Time
begin = '2012-09-01'
end = '2020-01-01'
dt_start = datetime.datetime.strptime(begin, '%Y-%m-%d')
dt_end = datetime.datetime.strptime(end, '%Y-%m-%d')
one_day = datetime.timedelta(1)
start_dates = [dt_start]
end_dates = []
today = dt_start
while today <= dt_end:
tomorrow = today + one_day
if tomorrow.month != today.month:
start_dates.append(tomorrow)
# End Date is 1 millisecond before midnight of the first day of the next month
end_dates.append(today + datetime.timedelta(1 - 1/86400/1000))
today = tomorrow
end_dates.append(dt_end)
start_times = Time(start_dates)
end_times = Time(end_dates)
start_times = start_times[:-2]
end_times = end_times[:-2]
for i, j in zip(start_times.isot, end_times.isot):
print(i, j)
start_times and end_times are both astropy Time objects that can then be used to filter and appropriate split the input observations.
The loop prints the following:
We are working to support multiple different datasets, however, as it stands indexing multiple datasets places them all in the same set of binary files. Ideally, we'd want to be able to subsample the data so we can combine different dataset indexes or split them apart trivially.
The solution would be to separate each dataset's observations into their own subdirectory and then separate each binary file by calendar month. This way downsampling the full dataset becomes a simple table query.
Proposed schema:
I've modified a code snippet from Stack Overflow that can be used to divide into calendar months (this could be a good starting place for any implementation).
start_timesandend_timesare bothastropyTime objects that can then be used to filter and appropriate split the input observations. The loop prints the following: