dotoritos-kim / dxf-json

perfect dxf parser
GNU General Public License v3.0
41 stars 2 forks source link

Building a large dxf file stream parser #3

Open dotoritos-kim opened 5 months ago

dotoritos-kim commented 5 months ago

Related issue

Currently, there is a size limit in the v8 engine, as shown in the issue link above.

To solve this, you need to create a parser that breaks the dxf file into chunks, reads them line by line, and parses them into json.

However, in order to create such a parser, the following problems must be solved.

  1. When cut into chunks, how should it be connected naturally to the next chunk?
  2. What to do in situation 1 when the data is recursive
  3. Enables reuse of existing parser source code

There is even a function to load the current file in chunks and save it as json. However, I think we need to discuss how to create a class that performs stream parsing while utilizing existing code.

dotoritos-kim commented 5 months ago

There is a way to implement streaming parsing functionality by reusing the existing parser.

As a result of talking with @Phryxia

Divide into chunks and proceed with stream parsing. It seems sufficient to slightly modify the DxfArrayScanner in such a way that when the last buffer of a chunk is reached while performing next or rewind, it gets the next chunk.