Automatic List Regeneration When Using Relative Date Rules

[pushchris]

First pass at allowing for daily regeneration of lists that include relative date math in them. This solution is relatively efficient for date math that compares forwards (or looks for dates greater than the generated value) but very inefficient for date math comparing backwards. This is because backwards comparison typically opens the criteria for membership in a list to a greater population requiring re-evaluating all users in a given project (not just looking at those already in the list). Depending on user base size this can take a solid amount of time.

Will continue investigating how to improve situations involving backwards looking relative dates, but most described use cases look forward to limit membership for only a defined criteria before removing users already in the list (vs looking to add more).

Addresses #514 and #535

[leobarcellos]

Testing it right now, it seems to be populating however it's slow, and it was running for 15min+ (We only have 16k users on this project)

I was checking errors and I saw these:

I've just rolled back. Please let me know if I can help you somehow.

[leobarcellos]

Update: it seems not related to this PR, but with one of these:

[williamhrs]

(about the relative data rules, it would be AWESOME if it would work also on the gate action inside the journeys. )

[pushchris]

(about the relative data rules, it would be AWESOME if it would work also on the gate action inside the journeys. )

@williamhrs relative dates should already work inside of gates since journey processing happens live (there is no pre computation unlike lists). If you encounter any issues please report them, but the logic as it stands supports them

[pushchris]

Update: it seems not related to this PR, but with one of these:

@leobarcellos thanks for the catch, this has been addressed in a separate PR

[williamhrs]

(about the relative data rules, it would be AWESOME if it would work also on the gate action inside the journeys. )

@williamhrs relative dates should already work inside of gates since journey processing happens live (there is no pre computation unlike lists). If you encounter any issues please report them, but the logic as it stands supports them

Oh so it already supports handlebars! I actually didn't try it. Awesome

[pushchris]

@leobarcellos have you had a chance to try this PR with the fix for the issue you reported? Would be curious to hear how its working for you!

[tanmay-predisai]

@pushchris - I am just curious about this, when are you planning to push the next release? ( I was looking forward to the Performance improvements, this one, and hence the question :) )

[leobarcellos]

@leobarcellos have you had a chance to try this PR with the fix for the issue you reported? Would be curious to hear how its working for you!

Just updated it. This project have 24353 users and I created two lists using same segmentation (email_clicked created at is on or after now - 1 days) -- one using dynamic date, and the other one hardcoding date.

It was "Loading" for 7 minutes, and then showed up 1770 users on the dynamic date one. However for some reason it just showed up 612 on the hardcoded one, weird. It's stuck on loading, I guess we are having another issue because we are always seeing 5k+ jobs in queue on performance view, something's wrong

Well, at least about this specific PR, seems like it's working, I just wanted to do a final test comparing a hardcoded date vs dynamic date. I will post here if I can manage to get to work.

[leobarcellos]

This error below is new for me, never noticed it. Maybe it's related to this PR. It was on RuleService.ts but seems like it wasn't a changed part on this PR.

I'm printing it here, maybe it's useful for you:

Looks like it's trying to read "name" from undefined, but it doesn't make much sense since the event object do have an "name" attribute. Like I say a lot here with my team: "something wrong isn't right xD"

[williamhrs]

We're seeing some lists, even the ones that aren't dynamic/relative getting stuck on the "loading" state.

[pushchris]

@leobarcellos That is a bizarre error but most likely not related to this PR. If you are generating your own images, could you throw a console log to see what dbEvent is on line 46 of EventPostJob? Would be curious what data is coming back, based on the code it shouldn't be possible for the name to not be present