Event formatter: sandiego.org dates

[mnich0ls]

Create a service to convert event dates for events scraped from sandiego.org. Event dates should be stored in ISO 8601 (YYYY-MM-DD) and the original date should be saved as an attribute on the event called original_date.

Some dates that have been scraped are in a date range formate, e.g. Feb 12 - ongoing or Feb 21 - Jul 21, 2019

For now, when this service runs to convert dates, dates that are originally formatted in the range format should be set to the current date when the current date falls within the given range e.g. if the service runs on March 13, 2019, when it processes an event with the date range: Feb 21, 2019 - July 21, 2019, then the date property for the event should be set to March 13, 2019 and another property should be added to the event called end_date and set to July 21, 2019

For events with an "ongoing" date range e.g. Feb 12 - ongoing, set the end_date property to "ongoing"

For events that have a date range in the future (i.e. the current date doesn't fall within the date range) , the date of the event should be set to the first date of the range. For example, assuming the current date is: March 12, 2019 and an event has the date range: April 11, 2019 - July 3, 2019, the date of the event would be set to: April 11, 2019 and the end_date would be set to: July 3, 2019

[jremi]

Just to clarify ISO 8601 date format would be YYYY-MM-DD

For example, September 27, 2020 is represented as 2020-09-27.

Is this correct?

[mnich0ls]

Yep, that's correct.

[jremi]

Here is an edge case:

For example: The events date shows Apr 6 - Apr 7, 2019 This would be considered based on your comment

dates that are originally formatted in the range format should be set to the current date

In this example above we probably do not want to replace the Apr 6 with the current date of March 11 since the events date has not started yet.

Does this make sense? What I am saying is that you would not want to replace the date with the current date if the date of the event has not yet occurred.

Please clarify

[mnich0ls]

You are correct, we would not want to replace that date with the current date. I will update the issue to clarify.

[jremi]

Please review this gist... It contains my logged test results of the new code I wrote to process the scraped event data and update all the dates accordingly.

https://gist.github.com/jremi/360a10b70003729f925604c789f1d6c9

In addition: I assume we should just add this to the regular scrape process. This way it just automatically runs this format date method each time it is going to push the data to Firebase storage.

Let me know.

Bid $100

[mnich0ls]

Excellent!

I would prefer that it be a separate process so that any future changes that need to be made to date formatting are easy to make without touching the code that does scraping, but since you wrote the scraper, I defer to you on how you would like to organize it.

I’m good with $100. This will probably be the last task for this month.

Thanks, Mike

On Mar 12, 2019, at 12:04 PM, jremi notifications@github.com wrote:

Please review this gist... It contains my logged test results of the new code I wrote to process the scraped event data and update all the dates accordingly.

https://gist.github.com/jremi/360a10b70003729f925604c789f1d6c9

In addition: I assume we should just add this to the regular scrape process. This way it just automatically runs this format date method each time it is going to push the data to Firebase storage.

Let me know.

Bid $100

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[jremi]

Sounds good.

I just finished migrating this new module into the scraper. What I did was create a new subfolder called addons and inside there I added a new module called EventDatesFormatter. The scraper is now requiring this module and calling it to have all of the events updated with the proper ISO date format / properties.

Question: If the event does not have an end_date value do you want me to still add that property to that object and set the value to empty string? or do you want me to just not include that property on the event?

For the sake of uniformity, I suggest we set the end_date to empty string if the event has no end_date. Let me know.

[jremi]

I just ran this new module against your Firebase.... you should be able to see the new updated properties on each object.

• original_date [Added] - (Contains the original data from event.dates)
• dates [Modified] - (Contains the start date of event in ISO 8601)
• end_date [Added] - (Contains the end date of event in ISO 8601)

[mnich0ls]

Awesome. Regarding your question about whether to set an ‘end_date’ property for events that don’t have an end date or range, I think it would be best to set the property and have it be null.

If we do that, it probably makes sense that the ‘date’ property be changed to ‘start_date’. That way all events have a ‘start_date’ and ‘end_date’ property, where a single day event would have a null ‘end_date’.

On Mar 13, 2019, at 6:41 AM, jremi notifications@github.com wrote:

I just ran this new module against your Firebase.... you should be able to see the new updated properties on each object.

original_date [Added] - (Contains the original data from event.date) dates [Modified] - (Contains the start date of event in ISO 8601) end_date [Added] - (Contains the end date of event in ISO 8601) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[jremi]

Please change status of this issue to "in process"

[jremi]

Firebase will not store a value in the database if you set the value to null. I attempted and it will not work. It has been suggested on StackOverflow to set the value with 0 which can be considered falsey value. For now I am setting end_date as 0 if the event is single day.

Please Venmo and then I will commit, create pull request and we can mark this issue as closed.

[jremi]

Pull request created https://github.com/mnich0ls/evee-sd/pull/14

[jremi]

JSDoc Reference Doc: JSDoc