Offering assisstance

[arekbulski]

I am the current developer of Construct library. I took it over in 2016 and since then, the project has flourished to a point where its pretty much done and perfect. I am looking a for a new project to participate in or even take over if needed.

I would ask that you point me to what I can do for the project. You can also consider giving me write access to repository.

[GreyCat]

That's perfect!

Currently, we have a short term goal — publishing v0.8, which is more or less ready to ship.

After that, our mid- and long-term goals could be broadly separated into several categories:

• Infrastructure
  • Overall project infrastructure overhaul — for example #62 (better, modular CI)
  • Per-language improvements — publishing and maintenance of individual languages' modules, improving per-language documentation, stuff like better C++ support on various platforms, etc
  • Finalizing and getting everyone to use so called "big format model" — #165
• Documentation and PR — helping others find out about Kaitai Struct, more articles on using it, various introduction courses, more "cookbook"-style entries into the docs showing various approaches to more complex problems, etc, etc.
• Various improvement ideas for the compiler and code generation. They are generally all in our issue tracker, filed as enhancements. Some are relatively small, some are major, such as:

  • 27 - serialization support

  • 229 - our own YAML parser

  • 69 - optimal operator parenthesis generation

  • 44 - better support for substreams (right now we do substreams "manually", by copying bytes into in-memory array, this works, but is not super efficient)

Tell me what you have in mind and what are you more inclined with?

[arekbulski]

I would be willing to check these first:

• writing easy to follow tutorials before cookbook recipies, Construct has a superb tutorial imho
• Python code generation overall
• test suite, Construct has a huge test suite that made it close to bug free

Something that has to do with me being Construct developer:

• getting some features (classes) into Kaitai that Construct has
• getting high performance to Construct that Kaitai has
• I had and used substream implementation myself but dropped it because performance is not even a goal in Construct, believe it or not, it doesnt not even have any benchmarks. I will show you the implementation after I dig it up from the repo archive

The whole reason why I looked at Kaitai was because I read this article. Author was very generous in words towards Construct for its tuorials but its performance was even below feasibility. https://pythonistac.wordpress.com/2017/03/09/python-network-packet-dissection-frameworks-shootout-scapy-vs-construct-vs-hachoir-vs-kaitai-struct/

Bottom line is, I would like to improve Construct by getting its performance closer to Kaitai and at same time improve Kaitai by getting its features closer to Construct. Currently I am working on cythonizing Construct to improve its performance (substantially, 10-100x goal). Thank you very much for your breakdown post. Its very useful and I appreciate it.

[KOLANICH]

104

[arekbulski]

104 made me laugh, thanks Kolanich. :laughing:

[GreyCat]

writing easy to follow tutorials before cookbook recipies, Construct has a superb tutorial imho

I guess you're right, you're more than welcome to take a shot ;)

Python code generation overall

Sure, that would be nice to have someone who's proficient in Python to take a look :)

test suite, Construct has a huge test suite that made it close to bug free

We also have fairly large test suite that we support for all languages: http://kaitai.io/ci/ There's also some long-term plan to implement language-neutral way to write assertions that should make it much easier to implement new languages support for KS.

I had and used substream implementation myself but dropped it because performance is not even a goal in Construct, believe it or not, it doesnt not even have any benchmarks.

Cool ;) But remember that we'll need that not only for Python, but for all other 10 languages as well? ;)

The whole reason why I looked at Kaitai was because I read this article

I've seen that article, and I'm not so sure about benchmarking methods that author used — i.e. why Python, why not multi-threaded, etc, etc?

[arekbulski]

I am currently working on Construct (cythonizing it), so it should take me a week or two. Then I would like to do the same to Kaitai. Posted the proposal in #311 .