Event log creation takes a long time to complete

[felipecr00]

Hi everyone!

I have a Dataframe of about 15Million-rows. I have correctly defined the case_id, activity, timestamp, among others. My problem is that for large datasets it takes too long to run, can even reach up to hours.

Is there any way I can run this transformation fast enough ?

Thank you! Felipe

[fmannhardt]

Try to add validate = FALSE to the call when creating the event log: https://github.com/bupaverse/bupaR/blob/5fcb7540be4c0f166f352842bbe0810245594932/R/eventlog.R#L61

This disables certain consistency checks that take quite a while on large logs.

[felipecr00]

It works!, this reduces the log creation time considerably. Just to be clear, what does the deactivation of validation affect?

Thank you!

[fmannhardt]

The validation checks (the list is not complete):

• activity instance ids are unique across cases
• activity instance ids are not connected to multiple activities
• not multiple resources per activity instance id

You can see the code here: https://github.com/bupaverse/bupaR/blob/5fcb7540be4c0f166f352842bbe0810245594932/R/eventlog.R#L224

Problem with this validation is that it is not really efficient. We left the parameter on TRUE by default for backwards compatibility. I think it should be better documented that this is really a showstopper for large event logs.

[fmannhardt]

@gertjanssenswillen I patched the website documentation here: https://github.com/bupaverse/website/pull/1