search

taocpp / PEGTL

Parsing Expression Grammar Template Library
Boost Software License 1.0
1.94k stars 228 forks source link

Question about discarding buffer #173

Closed Shaheen47 closed 5 years ago

Shaheen47 commented 5 years ago

Hello, when using stream input I have to pass give maximum value in the parse function. I am a little bit confused about this maximum value and difference between it and the chunk size. Could you please give me more details about that.

ColinH commented 5 years ago

In a nutshell, the maximum size is how much the grammar expects to read between calls to discard(), while the chunk size is something that the PEGTL adds to that value to determine the effective size of the buffer in order to have a guaranteed minimum size. This area will be better documented in the future, though some detail might still change...

Shaheen47 commented 5 years ago

Thanks for the answer. I have another question regarding this. When discard() is called, what exactly happens to the buffer? the buffer will not contain exactly the data that has been pared when discard is called .it will contain more data.so what happens here, is all the parsed data deleted and the data which not parsed yet still there and another data is added to it until the buffer is full? or something else happens here? I need to understand what happens here in order to define the minimum maximum value that works with my stream parser.

ColinH commented 5 years ago

When discard() is called, you should assume that all pointers into the buffer are invalidated because any remaining data that has not been consumed will probably be moved to the beginning of the input's buffer space. That is 'probably', because IIRC the implementation currently doesn't always move, if you consumed less than the chunk size then the data stays where it is.

Shaheen47 commented 5 years ago

I would have further questions regarding this issue soon.so I will leave it open. Thanks again for the help.

ColinH commented 5 years ago

No problem, happy to answer all questions.

ColinH commented 5 years ago

Actually, I'll close this for now since the initial issues are resolved, just so that it doesn't show up as something we need to actively work on. You can still post related questions here, and we will still get notifications for them, and answer...