Closed BrandonDR closed 5 months ago
For context, this seems to related to the fix for #340, specifically this commit https://github.com/dseiler/docx-templates/commit/8eeec1fa0227813b1787ef0e8d87edf3a94fcf29
Hi @BrandonDR , great idea to add this. The hardcoded limit seems a bit low, now I think of it. I can totally imagine legit use cases hitting that limit, so making it configurable makes sense to me. What is the limit value are you using in your code? Maybe we should increase the default value too, in addition to making it configurable.
Hi @jjhbw,
I am using the Infinity
constant, which disables the exception but this would not be a good default if there could still be an infinite loop.
const buffer = await createReport({
//...
maximumWalkingDepth: Infinity,
//...
});
I am opting for a 10 minute process timeout instead. As I am spawning a node process from my PHP queue worker and handling the exception case there.
Our use case has a bunch of varied templates that we are building all the time for a range of clients, so we need something to accommodate a wide range of document sizes and templates. For the case we encountered this error there was a FOR loop with a 20 column table with ~2400 rows which managed to hit this limit. This would be an extreme outlier.
Hard to say, maybe 10 million might be reasonable headroom. There is a trade off of how quickly the library should give up.
Maybe a timeoutSeconds
option with a setTimeout approach might more sense?
Excellent, thanks a lot for contributing!
@BrandonDR thanks for this update. Yes, it makes sense to have this limit configurable. I set is initially to 1 Million because with 1 Million it takes roughly 10 seconds to hit the limit. In my documents I usually never get close to this limit (even with quite complex docs with many loops etc.). But of course there could be use cases where you need more (however a docx with more than 2000 rows might become unhandy ;-) ). My biggest concern was the memory consumption. If it exceeds the 1 Mio loops (which is usually an indication that something went out of control), the memory consumption on our cloud infrastructure might get critical.
@BrandonDR quick addition: if the users can create the templates themselves they could always crash the server with a statement like this:
{{! counts = [0];}}
{{ FOR count IN counts }}
{{! counts.push(-1);}}
{{ $count }}
{{ END-FOR count }}
Besides that there could be still some very rare situations in a template where the engine goes into an infinite loop. So in production I would always set this option to a decent value to prevent crashes.
if the users can create the templates themselves they could always crash the server with a statement like this:
Just a pedantic side note: be aware of the security considerations when letting users upload templates that are executed on your servers. You are effectively executing user-provided javascript in that case. See https://github.com/guigrpa/docx-templates?tab=readme-ov-file#performance--security
@jjhbw @dseiler Good notes, I am running this from separate 'queue workers' so memory usage is not a huge concern in regards to crashing the whole application. It is reasonably well isolated but I should isolate it further regarding security - it should be on a dedicated vm for untrusted user uploads with minimal permissions/config/env vars etc. I have thought of this prior. But of course there are deadlines to meet. Thanks for the reminder
This PR implements a new option
maximumWalkingDepth
. This option replaces the hard-coded1000000
(1 million) max loop iterations. This loop counting is intended to avoid an infinite loop. I have retained the default behaviour but updated the InternalError message to 'infinite loop or massive dataset detected'. This should make this issue easier to debug.I have added TS doc comments and updated the README to document this option. I have also added a
Tip:
to suggest the use of theInfinity
constant which will disable the infinite loop detection. I intend to use this in combination with a timeout on my node process instead of an arbitrary limit.Motivation
When generating a document using a FOR loop with 2000 iterations and each loop inserting (INS) many values, I was able to exceed 1 million loops in the walkTemplate function. This caused a generic 'something went wrong with the document. Please review and try again' error
I appreciate your work on this library!