Holiday filter issue

[alexveden]

When I was analyzing the #101 issue I've found that contracts still contain holidays for example US.F.GC.G18.180226:

My investigation showed that the holidays are filtered in the DB, but actually the problem in PST timezone. We store all dates in UTC timezone, and the date 2017-12-26 is considered to be a business day (at least by the current holiday calendar algorithm).

So basically we have 2 ways to avoid this issue in the future:

1. Include holiday filters straight into continuous futures quote build algorithm
2. Switch timezone of all quotes from the PST to EST.

[spickering-git]

@alexveden possible solution ideas

Solution A

look at dates when holidays, only start collecting data on the day after a holiday 00:00 UTC + 8. The current data collection that is filling V2 from CQG data will be shutting off soon, so this will need to be done in iqfeed data collection.

Solution B

add the day after a holiday also as a holiday. Would there be a potential knock on effect from not trading for 2 days?

Do we need to re-run or re-calibrate after making this change?

[alexveden]

I think that Solution A is the quickest way to temporary fix this issue.

But if we want to avoid half-measures, I think we will need to add this functionality to Continuous Futures engine or even in the DataManager engine. Because of other markets like European market, have different holidays set, and it's not an elegant way to handle holidays inside the datafeed part.

[spickering-git]

@alexveden We are getting Not Aligned messages on every alpha. This is probably related to the holiday on Dec 25. https://10.0.1.2:8888/notebooks/smart_campaign_generation/Top%20Select%20dashboards/Div_Futures_RR_PlusP_C_spreads_SmartC_Div_Str_Top_select_V5_Dec27_2017.ipynb

Do we need to re-run anything to fix this?

[alexveden]

We are getting Not Aligned messages on every alpha. This is probably related to the holiday on Dec 25.

Correct, I've fixed this in IQFeed script already.

Do we need to re-run anything to fix this?

I don't know. If you concerned only about the alignment warnings, why will be automatically expired in 7 days.

[alexveden]

I've implemented the hard-coded holiday filter for US market using PST timezone, but as I see you are going to use EU markets. EU holidays will require another patch (half-measures), or special holidays handling in the framework (optimal way).

[spickering-git]

@alexveden We had a significant problem today where there is no SPAN file published on Jan 1 2018. The last SPAN file loaded was for Dec 29 2017. This resulted in an error in the model where none of the EXOs were properly calculated. All EXOs and Alphas returned 'EOD quotes data delay'.

[alexveden]

It should be naturally fixed when SPAN data update arrives next day.

[nikolas-joyce]

But the position will not be valid until execution time tomorrow when span arrives and the positions are calculated?

Sent from my iPhone

On Jan 2, 2018, at 6:40 PM, alexveden notifications@github.com<mailto:notifications@github.com> wrote:

It should be naturally fixed when SPAN data update arrives next day.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftrendmanagement%2FTmqr-framework-2%2Fissues%2F102%23issuecomment-354926887&data=02%7C01%7C%7C3cd0b52476bd4d7d73f008d55253618f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636505440420062531&sdata=vtMZO75QgdODT8w5cbEPMff0Jsm9T6J6poUZymnTSTQ%3D&reserved=0, or mute the threadhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FARobLM88bldeeadkHCds9bz_yWsZdVaVks5tGuipgaJpZM4RMfri&data=02%7C01%7C%7C3cd0b52476bd4d7d73f008d55253618f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636505440420062531&sdata=fXKARylUD79%2BVt4L%2BCzEi2EhkBwwHF9d8TVjA%2F6wg1k%3D&reserved=0.

[alexveden]

yes, the position will be invalid until next execution time

All the best, Alex среда, 03 января 2018г., 07:16 +04:00 от NikolasJoyce notifications@github.com :

But the position will not be valid until execution time tomorrow when span arrives and the positions are calculated?

Sent from my iPhone

On Jan 2, 2018, at 6:40 PM, alexveden <notifications@github.com<mailto: notifications@github.com >> wrote:

It should be naturally fixed when SPAN data update arrives next day.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub< https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftrendmanagement%2FTmqr-framework-2%2Fissues%2F102%23issuecomment-354926887&data=02%7C01%7C%7C3cd0b52476bd4d7d73f008d55253618f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636505440420062531&sdata=vtMZO75QgdODT8w5cbEPMff0Jsm9T6J6poUZymnTSTQ%3D&reserved=0 >, or mute the thread< https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FARobLM88bldeeadkHCds9bz_yWsZdVaVks5tGuipgaJpZM4RMfri&data=02%7C01%7C%7C3cd0b52476bd4d7d73f008d55253618f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636505440420062531&sdata=fXKARylUD79%2BVt4L%2BCzEi2EhkBwwHF9d8TVjA%2F6wg1k%3D&reserved=0 >. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

[spickering-git]

can we have it refer back to the data from Dec 29, the last valid date of data that way we won't have a failure of the model where all of the positions are prompted to come out because of a missing piece of data.

[alexveden]

can we have it refer back to the data from Dec 29, the last valid date of data that way we won't have a failure of the model where all of the positions are prompted to come out because of a missing piece of data.

The only way to get rid of this is to delete all records for Jan 2 in quotes collections and rebuild all indexes and alphas. But this is very complex and risky operation, are there any alternatives?