Open lukastribus opened 3 years ago
Hi Lukas! I hope this answer can help to your analysis.
Is that the expected behavior in FORT? It is one-process with multiple threads, so a crash would achieve this, correct?
Correct. A crash will cause FORT validator to stop, which means that also the RTR server will go down. The key point here is: when will FORT validator crash? Definitely due to bugs (hopefully, shouldn't be much of these, but nobody's perfect) or a programming error (logged at crit
level, see Logging#level at our docs).
Regarding the other issues:
http.connect-timeout
, http.transfer-timeout
, http.idle-timeout
, rsync.arguments-recursive
, rsync.arguments-flat
). By default, once the connection is established FORT validator will try to fetch all the required data from the endpoint; if data is being transferred, the connection won't be killed, so FORT will wait until the data is fully transferred (unless the connection is terminated by a local network or remote issue). In other words, the RTR server won't die even if the validation takes a while to complete.I'm also thinking about monitoring (maybe without regex'ing logfile)
- how to best monitor for periodic successful validation runs
- how to monitor validation run time
Oops! As of today this data is logged at a level info
in the operation logs (you'll have to set it before running FORT, using --log.level=info
, since the default level is warning
).
Is there some stat socket that we could query to check for things like last validation start time and last validation completion time?
This will definitely be at our TODO list, so for now "regex'ing logfile" (using the info
level) is the way.
Regarding glibc memory allocation: when using glibc, should we just use MALLOC_ARENA_MAX=2 always or only in environments with limited memory? If this is a good middle ground, I'd prefer to just use it always in the glibc world and have systemd unit files set this.
Yes, I would recommend to use it in environments with limited memory. Of course, there's no problem using it always, since its main goal is to help.
Thanks for the feedback.
I think a stat socket or better yet a HTTP endpoint with a REST interface or something, that returns general health and validation metrics (especially last validation run - start, time and stop) would indeed be important for active monitoring.
I agree, this will be a nice (and likely needed) feature.
Newer version v1.5.0 will try to attend some of the points stated at this issue.
Regarding monitoring, I will build a rtrdump based tool to check for stalled RTR endpoints (same RTR serial and data output after X amount of time = monitoring alert). I believe this is a better way to monitor validator/RTR server health than relying on validation timestamps from a HTTP API.
Slightly off-topic: do you consider doing garbage collection based on the files not referenced in valid current manifests as opposed to rsync --delete/RRDP withdraw) ?
Thanks
I believe this is a better way to monitor validator/RTR server health than relying on validation timestamps from a HTTP API.
I agree, that's a good approach to monitor RTR server health.
Slightly off-topic: do you consider doing garbage collection based on the files not referenced in valid current manifests as opposed to rsync --delete/RRDP withdraw) ?
Well, I've read it a moment ago. Seems a good suggestion but I haven't discussed it with the team yet, so we need to analyze it in order to take a call on what to do.
Status:
crash bugs in the validation code
As has been previously mentioned, Fort panics when it detects programming errors. Because the validator and RTR server are part of the same binary, validator errors also bring down the RTR server with it.
This has worked this way since the inception of the project.
hangs during RPKI validation (even in rsync), that block the entire validation
There are a few timeouts in place (1, 2, 3, 4, 5), but I still believe the implementation to be naive.
My lead concerns right now are a timeout to rsync invocations, as well as a timeout to the overall validation. After that, I would like to worry about researching whether it's possible to assign timeouts to I/O operations in the cache.
Additional ideas welcomed.
memory allocation failures (failed malloc)
As of 1.6.0, Fort generally panics on memory allocation failures. As you proposed, this is intended to prevent Fort from advertising incomplete information, regardless of what the environment thinks is an adequate response to a failed allocation. All malloc
s outside of the asn1 code have been already wrapped.
I still consider this an ongoing effort however, because of the still pendig asn1 review, and also because some of Fort's dependencies sometimes obfuscate error causes. I don't know if there's a solution for the latter, other than ditching the dependency entirely.
I'm also thinking about monitoring (maybe without regex'ing logfile)
Embarrassingly, this is still meant to be addressed through the logs.
A Prometheus endpoint has branched off into issue #50, and I believe is the problem I will address next. The missing stats server is not only crippling production monitoring, but also profiling during development and testing.
So, in summary... not a whole lot of progress, yet. But this is rapidly becoming the lead of my worries.
Hello,
I'm currently evaluating the FORT validator and have a few questions.
I'm concerned about bugs, misconfigurations or other issues (in all RP/RTR setups, not specific to FORT) that will cause obsolete VRP's on the production routers, because I believe this is the worst-case in RPKI ROV deployments.
I worry about how issues like:
impact the RTR service:
The best-case scenario for me is that the RTR server goes down completely and all RTR sessions die, so that the production routers are aware there is a problem with that RTR end point and stop using it (failing over to other RTR servers, if available).
Is that the expected behavior in FORT? It is one-process with multiple threads, so a crash would achieve this, correct?
I'm also thinking about monitoring (maybe without regex'ing logfile)
Other than parsing strings from logfiles, how could we best achieve this? Is there some stat socket that we could query to check for things like last validation start time and last validation completion time?
Regarding glibc memory allocation: when using glibc, should we just use
MALLOC_ARENA_MAX=2
always or only in environments with limited memory? If this is a good middle ground, I'd prefer to just use it always in the glibc world and have systemd unit files set this.