erosb
commented
5 years ago Hello @jklukas ,
thanks for getting in touch. In cases when there are "too many" validation failures the collecting failure mode can crash by its nature. Even if we optimize the failure propagation process, it can still fail, we can only adjust the treshold. This is why the fail-early mode has been introduced for production environments.
One thing that may be useful is to have an "intermediate" mode, which is collecting the failures, but can have a limit of maximum number of failures collected, so users would have a reporting mode which is more verbose then fail-early but still has a safety net to avoid memory problems.
jklukas
This is probably not easy to take action on, but wanted to at least make you aware of the failure mode. I saw a service die due to GC thrashing that eventually traced back to this library attempting to collect a large number of validation failures. It may be that the current method of calling
ValidationException.prependto create a new ValidationException for each failing node becomes overly expensive in terms of creating new objects.Unfortunately, I was not able to recover the JSON payload that triggered the behavior, but the service was able to start up cleanly and continue working after I switched on the
failEarly()option of the Validator.Feel free to go ahead and close this if you don't feel like there's any follow-up that can be done without having the JSON document that triggered the behavior.
Originating issue: https://github.com/mozilla/gcp-ingestion/issues/374
Stack trace
Processing stuck in step parsePayload/ParMultiDo(DoFnWithErrors) for at least 05m14s without outputting or completing in state process at java.lang.Throwable.fillInStackTrace(Native Method) at java.lang.Throwable.fillInStackTrace(Throwable.java:783) at java.lang.Throwable.