[Feature] Octolapse Integration

[FormerLurker]

This isn't really a feature request, but I have been comparing and contrasting our methods for extracting slicer settings. Mine seems to be faster (0.0054 vs 7.5 seconds), but yours is much simpler and more elegant (regex vs key lookup per slicer) and requires little knowledge about the kinds of settings that might appear. My routine extracts and formats the values a bit better. Here is some sample output of your routine:

  "GPXconfigOverride": "r2",
  "aboveRaftSpeedMultiplier": "0.3",
  "adjustSpeedForCooling": "0",
  "allowEaxisZeroing": "1",
  "applyBridgingToPerimeters": "1",
  "applyToModels": "20mm-box,0.5mm-thin-wall,planter_mini_middle_eastern_villas",
  "applyToolheadOffsets": "0",
  "avoidCrossingOutline": "0",
  "avoid_crossing_perimeters": "0",
  "baseProfile": "Prusa i3",
  "baudRateOverride": "250000",
  "bed_shape": "0x0,250x0,250x210,0x210",

and a similar segment from my routine:

"avoid_crossing_perimeters": false, 
        "bed_shape": {
            "front_left": {
                "x": 0.0, 
                "y": 0.0
            }, 
            "front_right": {
                "x": 0.0, 
                "y": 0.0
            }, 
            "rear_left": {
                "x": 0.0, 
                "y": 0.0
            }, 
            "rear_right": {
                "x": 0.0, 
                "y": 0.0
            }
        }, 
        "bed_temperature": 60.0, 
        "before_layer_gcode": ";BEFORE_LAYER_CHANGE\\nG92 E0.0\\n;[layer_z]\\n\\n", 
        "bottom_solid_layers": 4, 
        "bridge_acceleration": 1000.0, 
        "bridge_angle": 0.0, 
        "bridge_fan_speed": 100.0, 
        "bridge_flow_ratio": 0.95, 
        "bridge_speed": 20.0, 
        "brim_width": 0.0, 
        "clip_multipart_objects": 1.0, 
        "complete_objects": false, 
        "cooling": true, 

I'm now looking for ways to merge the simplicity and elegance of your method with the speed and output quality of my routine. For example, my routine requires that each setting is pre-defined like so:

GcodeSimplify3dSettingsDefinition('version', 'G-Code generated by Simplify3D(R) Version ', Simplify3dParsingFunctions.strip_string, ['version']),
GcodeSimplify3dSettingsDefinition('process_name', 'processName,', Simplify3dParsingFunctions.strip_string, ['misc']),
GcodeSimplify3dSettingsDefinition('apply_to_models', 'applyToModels,', Simplify3dParsingFunctions.parse_string_csv, ['misc']),

This would need to be maintained, obviously, which is yet another downside of my method. My routine also can search starting at the end or beginning of a gcode file, and stops after all settings have been found (which is why it's so fast even when used on huge gcode files).

Anyway, once I am finished with my code I'll post it and will send you a link. Also, thank you so much for posting your code and giving me ideas! It's just great :) I am so glad that this community exists so that ideas may mingle and grow stronger!

[tjjfvi]

@FormerLurker:

Yes, right now my plugin does zero post-processing to the slicer settings, though I'm definitely willing to change that.

Regarding speed, there are a couple changes I can/should make to speed up the process; could you send me the gcode file you used to test?

[FormerLurker]

@tjjfvi, it's just a really big gcode file (simplify). If you REALLY want it I can post it somewhere :) Basically I just picked a gcode file and ran it through both of our routines, making note of the execution time and output. I also test with Slic3r, which appears to prepend the settings to the start of the gcode file.

Also, I'm looking for a way to have one plugin install another during setup, which would allow us to do all kinds of interesting stuff. It's unfortunate that this is not currently possible (that I know of), else I would strongly consider implementing all of this stuff in your plugin, then having Octolapse call that to extract the settings. Currently I'm going to keep it separate, but I'd still love to collaborate! If you add a hook to your plugin I'll try to add an option to use it within Octolapse instead of the built in routine, at least until we can install one plugin from another.

I read that you just recently got your printer and decided to dive into plugin programming. That's pretty much what I did too!

[tjjfvi]

@FormerLurker:

Re installing one plugin from another, you can probably use a wizard for that (e.g. have the user click a button to send an API call to install the plugin.

That’s fine re the gcode file; I’m sure I can make one big enough :)

I look forward to collaborating with you in the future and hope we can eventually integrate our plugins!

[FormerLurker]

Hey @tjjfvi, I have my new settings extraction working now, though I've not tested every settings 100% to make sure I'm parsing it properly. I used your regexes and added some new ones to grab some other info (slicer version info, gcode date, etc). It's not nearly as elegant as your routine, but it delivers the data fully parsed for easy use, and is very fast. Due to the regex it is a bit slower than it was previously, but I think this design is much more flexible. You can tag the settings in this implementation, then filter them down to only search for settings you are interested in, which is nice for my purposes.

Check out this file from the new_settings branch of Octolapse: https://github.com/FormerLurker/Octolapse/blob/new_settings/octoprint_octolapse/gcode_preprocessing.py

Perhaps it will give you some ideas? Also, I wanted to ask if I could add that Cura template you posted to the Octolapse repository. I'd like to add a button that will display the template and allow users to easily paste it into Cura. I will be able to detect that the user is running Cura even without the template, so it could be in some kind of pop up suggestion with a link if Cura is detected but no settings are extracted.

Let me know if you have any questions about the code, especially if you have any suggestions for improvement!

[tjjfvi]

@FormerLurker: Neat; I'll take a look into the file you mentioned later. Does the analyzer run at upload or print start? Also, how could I view the output from your routine?

Yes, you can certainly use the template with Octolapse, just make sure to give credit 😃; I also created a small webserver to generate it (hosted here; source available here), if you'd like to always have an up-to-date version (defaulting back to the gist if desired). I will try to update the gist periodically, though.

[FormerLurker]

Of course I'll give you credit! How about this, which has been added to the Octolapse about page in the 'Octoprint development community' credits section (translated to markup for display, of course):

tjjfvi - created a Cura gcode settings template that can be used to extract slicer settings.

I will include a copy of the template in the repo with the following heading:

; *****************************************************************************
; This template is used with permission from tjjfvi (https://github.com/tjjfvi)
; Licensing information is available at
; https://github.com/tjjfvi/OctoPrint-SlicerSettingsParser/blob/master/LICENSE
; *****************************************************************************

How does that look?

[tjjfvi]

@FormerLurker: 👍 (unsure if reactions notify)

[FormerLurker]

I forgot to answer your questions!

Does the analyzer run at upload or print start?

At the start of the print. It's pretty unnoticeable.

Also, how could I view the output from your routine?

Try this code, which will search for settings in Slic3r Pe, Simplify 3d and Cura:

from octoprint_octolapse.gcode_preprocessing import GcodeFileProcessor, Slic3rSettingsProcessor, Simplify3dSettingsProcessor, CuraSettingsProcessor
import datetime
import json

def json_convert(o):
    if isinstance(o, datetime.datetime):
        return o.__str__()

def receive_update(percent_finished, time_elapsed):
    if percent_finished == 100:
        print ('Process completed in ' + str(time_elapsed) + ' seconds.')
    else:
        print ('percent finished: ' + str(percent_finished) + ' in ' + str(time_elapsed) + ' seconds.')

simplify_preprocessor = Simplify3dSettingsProcessor( search_direction="both", max_forward_search=1000, max_reverse_search=1000)
slic3r_preprocessor = Slic3rSettingsProcessor( search_direction="both", max_forward_search=1000, max_reverse_search=1000)
cura_preprocessor = CuraSettingsProcessor( search_direction="both", max_forward_search=1000, max_reverse_search=1000)

file_processor = GcodeFileProcessor([simplify_preprocessor, slic3r_preprocessor, cura_preprocessor], 1, receive_update)
target_file_path = 'PUT YOUR GCODE FILE PATH HERE!'
filters = None
# you can filter by tag by uncommenting and adding filters.  I have NOT updated all of the tags for each settings yet.
# filters = ['filter1','filter2']
results = file_processor.process_file(target_file_path, filters)

print (json.dumps(results, default=GcodeFileProcessor.json_convert, indent=4, sort_keys=True))

Note that I didn't add an 'unknown' settings category yet for settings that are not in my dictionary, but match a regex. I'll do this at some point. Also, I need to update the tags for each setting, which allows you to only grab a subset of all settings.

[FormerLurker]

one last thing. You need to install the file_read_backwards package via pip before the above will run. Hopefully I didn't miss anything else :)

[OllisGit]

Hi @tjjfvi , @FormerLurker ,

I was was looking for a solution to store all slicer-settings to my PrintJobHistory-Plugin

So I found this little helper here.

I am wondering why the size of the gcode file is relevant. From my understanding the metadata is located in top of the file. So just add a "metadata-end" marker to the file and stop parsing the following lines. I think the user could adjust this behavior...put everything in the start-gcode-section. If he didn't do that, the parsing takes a while...."blame yourself" Or is there something I missed?

BR Olli

[FormerLurker]

Well, the settings could, depending on the slicer, be at the beginning, the end, or both the beginning and the end of the file. With Cura there is some flexibility since it requires a settings extraction script to be added manually (note that multi-extruder settings are a bit of a pain to deal with. Let me know if you want to support these, because I've had a lot of experience with them recently, and there are a few bugs in Cura that have since been fixed but not released). For Slic3r/Slic3rPE and PrusaSlicer the settings are at the end of the file always, but some other info is at the beginning (Slic3r version and various extrusion widths for perimeters, first layer, etc). For Simplify3d all settings are all at the beginning.

Unless it has changed, @tjjfvi's method of extracting settings is both simple and elegant. It uses a few regexes and can extract most settings out of the box. I'm not sure if currently does this or not, but one could search for the first N and last N lines of gcode to find any settings that may be present, and the resulting settings can be extracted in string form.

The method Octolapse uses is somewhat more complicated because I need the values to be parsed into known types (ints, floats, bools, strings, csv strings, csv floats, etc...). The Slic3r routine searches the first 50 lines and the last 263 lines of gcode. The Simplify3d routine only searches the first 295 lines and nothing from the end of the file. The Cura routine searches the first and last 550 lines of gcode for settings since people may add the custom settings script anywhere in the start/end gcode. Once preprocessing is complete, Octolapse determines which slicer was used by returning the format that contains the most parsed settings. If you're curious about this, take a look at this file. Here is an example of creating a GcodeFileProcessor object, processing a gcode file, and determining which slicer settings were used:

https://github.com/FormerLurker/Octolapse/blob/a5489b8ce1b3d3ca533e8a4d24c69b9d99027ba6/octoprint_octolapse/settings.py#L580-L610

Anyway, no matter which method you use, you will either need to search the entire file for settings, or N lines from both the front and back in order to ensure that you extract every setting for Cura, Simplify3D and Slic3r/Slic3rPE/PrusaSlicer. If you decide you need settings fully parsed into the appropriate types, I'd be glad to help you figure out (and hopefully improve) the Octolapse extraction method. If you decide that the simple and elegant approach (i.e. the @tjjfvi way), I think that will be self explanatory. I'm excited to see what you come up with!

Edit: Not sure why that permalink isn't working.... Here is the code referenced in the link:

    def get_gcode_settings_from_file(self, gcode_file_path):
        simplify_preprocessor = gcode_preprocessing.Simplify3dSettingsProcessor(
            search_direction="both", max_forward_search=1000, max_reverse_search=1000
        )
        slic3r_preprocessor = gcode_preprocessing.Slic3rSettingsProcessor(
            search_direction="both", max_forward_search=1000, max_reverse_search=1000
        )
        cura_preprocessor = gcode_preprocessing.CuraSettingsProcessor(
            search_direction="both", max_forward_search=1000, max_reverse_search=1000
        )
        file_processor = gcode_preprocessing.GcodeFileProcessor(
            [simplify_preprocessor, slic3r_preprocessor, cura_preprocessor], 1, None
        )
        results = file_processor.process_file(gcode_file_path, filter_tags=['octolapse_setting'])

        # determine which results have the most settings
        current_max_slicer_type = None
        current_max_settings = 0
        if 'settings' not in results:
            return False
        settings = results['settings']

        for key, value in settings.items():
            num_settings = len(settings[key])
            if num_settings > current_max_settings:
                current_max_settings = num_settings
                current_max_slicer_type = key

        if current_max_slicer_type is not None:
            new_settings = settings[current_max_slicer_type]
            self.slicer_type = current_max_slicer_type

[OllisGit]

Hi @FormerLurker, thanks a lot for the detailed explanation! I didn't know that there is such a difference and the user is not able to influence this behaviour.

Your solution is really tricky..some lines from the top a few lines from the bottom....good idea!

To be honest, I didn't want to create my own implementation. Just using a ready to use plugin. At the moment it looks like that I could reuse this plugin and just add it to my dependency list and it is up to the user if he needs such information in the history.

It would be very interesting to know how big the files are in general. Do you think Gina has such information?

[FormerLurker]

You mean how large are gcode files in general? Mine typically run from 5MB to 100MB, but if you are outputting verbose gcodes (Slic3r) or using a fine layer height (0.05mm) they would be much larger.

I bet @foosel has some of this data. If not, it might be something interesting to add to the data collection!