When are you going to adopt Python 3?

[heck-gd]

Python 2.7 has been EOL for 3 years now. The version shipped in the generic python:2.7 image used here is from late 2019 and is vulnerable to critical code execution vulnerabilities such as CVE-2021-3177. I see this as quite problematic for a software used mostly in a security context, especially considering that it parses countless file formats from untrusted sources (not to mention that the celery processes doing so are running as root).

Fully adopting Python 3 would also reduce the kludge in parsers, where it's currently necessary to manually invoke the python3 interpreter if you want to use modern code/dependencies. Working with python2 parsers comes with its own sets of pains, e.g., anachronistic things such as unicode string handling, which is much more consistent in python3.

[salehmuhaysin]

hello there is a lot of security concerns with Kuiper not only vulnerable libraries (such as not proper authentication and authorization), if security is important i recommend to not connect it to enterprise network (use it in lab network and only allow specific users to access the lab network).

for the python3, I'm working on new release with complete rebuild from scratch with proper structure and new features, of course it is python3, currently I'm testing it and checking the licensing.

[heck-gd]

Hey @salehmuhaysin,

It's not directly related to this issue but I wanted to bring something to your attention that could be worthwhile to add to your new release before it's out (in case you are still using MongoDB and didn't change the collection layouts).

We noticed a critical issue with the MongoDB files collection. Since metadata about all files that belong to a machine is stored in a single document, large machines with around 30k files become highly problematic. The max document size in MongoDB is 16 MB and it cannot be increased without resorting to tricks.

So I put some work into restructuring the collection to represent each file as a single document. This has the following advantages:

• Document size limit is never hit.
• The flattened design allows all queries to be directly executed on the database without having to do processing/filtering on the Python side, greatly increasing performance in many scenarios.
• The old code has concurrency problems if multiple tasks are running for the same machine, e.g., updates can be lost. The new code does not have this problem because it expresses everything through atomic db operations.

You can find the changes here: https://github.com/GDATAAdvancedAnalytics/Kuiper/tree/mongodb-revamp - feel free to apply it to your code base or take inspiration from it. :)

I don't want to open a PR for this for Kuiper v2 because it's a breaking change for existing deployments. Theoretically it would be feasible to integrate it, but it'd require some fancy automatic migration mechanism for existing data.

[salehmuhaysin]

hello, There are a lot of issues with the structure of Kuiper, specially the mongoDB database and flask which make adding new features of fixing issue very hard. mongoDB is no longer in the new release since it is not efficient to do simple things like relationship and has other issues for scaling, the new release will be commercial in cloud with more features added including full API, user management, etc. it should be released soon.

still will continue the Kuiper as open source but the new one will be different since it is restructured from scratch

[salehmuhaysin]

Updates:

The Premium Version of Kuiper (CyberBomah) now officially released, some changes regarding this issue

• changed python 2.7 to python 3 with complete rebuild
• mongoDB no longer used.
• the release is on-prem not cloud (unless requested by customer)
• the mechanisms of parsing had a lot of enhancement to reduce memory utilization and improve concurrent processing.