Improve Efficiency for Large Files

[dHutchings]

Hello! I have found your preprocessing script very useful for my Flashforge Creator pro (I have 3 of them).

I typically print very large files on my CreatorPro2; typically, they take 1-2 days to pint.

Recently, I got a new computer, and after setting up PrusaSlicer and your extension; the exporting process would never finish.

The code was taking hours when it used to take ~2 minutes. I eventually tracked this problem down to a combination of Python versions (3.11 vs 3.10) and inefficient calling of regex.

I would like to submit a pull request with the following changes:

• Introduce a progress % shown on the screen.
• Improve the efficiency of the code by more judiciously using regex. Specifically:
• Remember when a regex has found a match. If a regex has found a match; do not keep running that regex for subsequent lines.
• Leverage the fact that most of the lines to be matched are typically at the very beginning or very ending of the file to more judiciously search for the right data. More specifically, some of the regexes search for data that is typically at the beginning of the file. Loop through the gcode file forwards, looking for only data at the beginning of the file. Then, loop through the gcode file backwards, looking only for data that is typically at the end of the file.

These changes put together make a substantial difference in runtime. For testing, I ran this code on a large gcode file "PanTiltBotCabling.gx.gcode", under various parameters. This file is rather large (53MB).

The original code, on a chromebook running python3.8, took 34 seconds to run and called regex 21.2 Million times:

python3.8 -m cProfile -s tottime ~/FlashForge-CreatorPro2-PS-Profile/post-processor/ff-creator-post-processor.py PanTiltBotCabling.gx.gcode
         26010583 function calls (26009787 primitive calls) in 33.912 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1   15.234   15.234   33.912   33.912 ff-creator-post-processor.py:14(<module>)
 21230873   13.087    0.000   13.087    0.000 {method 'match' of 're.Pattern' objects}
  2359063    3.448    0.000    5.121    0.000 fileinput.py:246(__next__)
  2359063    1.599    0.000    1.658    0.000 {method 'readline' of '_io.TextIOWrapper' objects}

The original code, on a chromebook running python 3.11, timed out after over two hours of runtime.

The new code, on a chromebook running python 3.8, took 10.5 seconds and (only) 4.7 Million calls to regex:

python3.8 -m cProfile -s tottime ~/FlashForge-CreatorPro2-PS-Profile/post-processor/ff-creator-post-processor.py PanTiltBotCabling.gx.gcode
100%
         7134459 function calls (7133663 primitive calls) in 10.492 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    4.433    4.433   10.492   10.492 ff-creator-post-processor.py:14(<module>)
  4719545    2.625    0.000    2.625    0.000 {method 'match' of 're.Pattern' objects}
     1029    0.862    0.001    1.410    0.001 {method 'join' of 'str' objects}
        1    0.848    0.848    0.898    0.898 {method 'readlines' of '_io._IOBase' objects}

The new code, on a chromebook running python 3.8, took 16 seconds and an identical number of calls to regex:

python3.11 -m cProfile -s tottime ~/FlashForge-CreatorPro2-PS-Profile/post-processor/ff-creator-post-processor.py PanTiltBotCabling.gx.gcode
100%
         7140131 function calls (7138972 primitive calls) in 15.957 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    8.486    8.486   15.957   15.957 ff-creator-post-processor.py:1(<module>)
  4719545    3.671    0.000    3.671    0.000 {method 'match' of 're.Pattern' objects}
      954    0.941    0.001    1.682    0.002 {method 'join' of 'str' objects}
        1    0.877    0.877    0.938    0.938 {method 'readlines' of '_io._IOBase' objects}

The root cause of this problem is inefficiency (many millions of calls to regex) and python 3.11 making many, many changes to regex, one of which presumably slowed this specific regex down greatly.

I have tested that this new code produces 100% identical results to the original code; at least for the prints I run.

Please consider this change! I hope this helps people out!

[Jacotheron]

@dHutchings Thank you very much for these optimizations.

When I wrote this, my primary objective was to get something that works. I am also very much beginner at python, so I am not surprised that my code may be inefficient. Most of my prints are typically a few hours, so waiting 20-30s for it to parse is still fine, but timing out after 2 hours, that is way to long.