remove min_date/max_date fields and have a single date field

hasadna / open-bus

:bus: Analysing Israel's public transport data

93 stars 29 forks source link

remove min_date/max_date fields and have a single date field #360

Closed OriHoch closed 2 years ago

OriHoch commented 2 years ago

following discussions on #359 and PR hasadna/open-bus-gtfs-etl#12 it seems there are too many problems and implementation complexity with the requirement to have min_date/max_date fields on the gtfs data

To reduce complexity we will remove this requirement for now

To Do:

change all gtfs DB tables - remove min_date/max_date fields and add a single date field
change gtfs-etl process to create a single record per date

OriHoch commented 2 years ago

assigning to @AvivSela and setting to In progress because it should be done as part of the currently in progress task to remove duplicate stops

OriHoch commented 2 years ago

@AvivSela I will implement the DB changes

AvivSela commented 2 years ago

so, are we going to duplicate the data for each date even it won't be modified? in case I uploaded stop, and tomorrow the stop is the same, should we upload it again?

OriHoch commented 2 years ago

yes, each record will be valid for a single date specified in the date field

OriHoch commented 2 years ago

I will try to have the DB changes ready today / tomorrow

OriHoch commented 2 years ago

DB changes are done and merged, I renamed all the tables to have gtfs prefix to make it clear these tables are only used for gtfs data - https://github.com/hasadna/open-bus-stride-db/pull/7

AvivSela commented 2 years ago

the changes in the GTFS ETL about the schema change are included in the pr (hasadna/open-bus-gtfs-etl#12) that was opened for #348