Three Questions

[geeksailor]

I have three questions about the Python salt-states and how they are written for the sake of pure curiosity.

1. I noticed that all the Python salts install to /usr/bin as root. Why is everything installed as root? Do they all need to be ran as root? Could they be installed as user in the future to say the ~/.local/lib/ so they don't give certain malware elevated privileges while running analysis?

2. Some of Python salts use git links to install from when some are on PyPi and at the same version or maybe even newer. Would it be more beneficial to pull from PyPi for installs rather than git? The one's I found were:

   • Unfurl (I believe they are different forks between the git and PyPi but the PyPi is a month newer on updates)
   • Ioc-parser(Git) vs ioc-parser-ng(PyPi) (Git was last updated in 2017 and PyPi in 2020)
   • virustotal-api(Git) vs virustotalapi3(PyPi) (PyPi is just 2 months older but both are still 3 years old since last update)

3. I ran a pip check I noticed that some packages overlap on dependencies for the Remnux project requiring different versions of some of the dependencies. I did notice some of the salts use virtualenv. Would it be useful to encase some of those as well so they can be split off and each use the specific version of the dependencies they need? I have been trying to look into if Pipx can be used with salt, but no luck so far on that. The conflicting dependencies are:

Conflicting dependency[dep it is required by(req vers)]

• PyYAML [docker-compose(>=3.10, <6.0), Qiling(>=6.0), python-fx (==6.0.0){needed for qiling}] NOTE: I think because docker-compose requires <6 and not =<6.0, it balks at installing 6.0)
• Unicorn [qiling(>=2.0.1), speakeasy-emulator(==1.0.2)]
• pillow [python-fx(==9.2.0), thug(==10.0.1), pytesseract(>=8.0.0)]
• python-magic [droidlysis(==0.4.12), thug(==0.4.27)]
• antlr4-python3-runtime [python-fx(==4.8), stix2-patterns(~=4.9.0)] NOTE: Not sure if stix2-patterns is used for Remnux but found it installed and conflicting. It is a dep of a dep of mwcp if this helps.

Thug, python-fx, and qiling seem to come up a lot and seem to be very specific on what it needs. I would say those may need to be put in a virtualenv so they can have the the specific dependencies they need.

I think the only programs i have had noticeable issues with so far with these dependency issues are between trying to use qiling and speakeasy. Their dependency requirements for unicorn breaks one or the other. I have not noticed any issues so far with any of the other dependency requirements but I will make submit a bug report if I do. But hopefully noting these conflicting dependencies might head off any possible future issues.

If you are in the U.S. have a very Happy Thanksgiving! Cheers!

[geeksailor]

Devs / @digitalsleuth (sorry bud to always tag you) Also just caught that docker-compose is at 1.29.2 from PyPi from where it is sourced for this project. But I guess Docker has been bad and has not been updating their PyPi since 1.29.2 was their update from May 2021. But if you go to their git https://github.com/docker/compose it was last updated a few days ago (Nov 23rd). You might need to consider switching sourcing on dependency from the PyPi to their Git in the future to ensure you are getting up to date versions of it. Cheers!

[digitalsleuth]

Hey, sorry I'm just now seeing this, not sure how I missed it :) I'll do my best to answer your questions in order:

1. Regarding the installation as root, the primary motive around this was to make sure the applications were available to all users on the system, and not just the current one. They don't all need to be run as root, but some of them may if they have to use system devices. Installing in the users local means they won't necessarily be available for any other users created on the system.

2. Some of the packages that are installed via git instead of Pypi (pip) are that way either because the Pypi version at the time wasn't up-to-date, or there is/was no Pypi package. I'll take a review of those we have installed by git and see if they can be updated to be pip installed.

3. I have been working on moving some of these tools to virtualenv's, but am also trying to make sure they'll work on Jammy for support. Some of them don't play well with virtualenv's, but they're coming along.

4. (Docker Compose) This is definitely something that can be looked at, but there are some breaking changes with the move to the newer version which I'll have to test. Ideally, it should be pretty easy.

Since there are a few things on the docket here, it might take a week or so to get to them, but I'll be looking at them this weekend (as well as peepdf).

Cheers!

[geeksailor]

@digitalsleuth All good. I wasn't worried so much about getting answers for the three questions as much making sure that the Docker Compose thing was noticed. But thank you for answering them. Hopefully you can grab a newer version (whatever is most stable) for the project to use and not something 2 years out of date (thanks for that Docker!). I may have spoken too soon on the Docker thing. https://github.com/docker/compose/issues/9571. Python was a part of their legacy v1 of their project which is no longer supported. V2 is what is on their Git which is now written in Go. Not sure if this changes things.

As for the other things: Running as root for all to be able to access the tools makes sense. I may play around with that on my own just to mess with local installing to see what I can find out. The Pypi vs git was just some things I noticed. I wasn't sure if installing from Git or Pypi was easier so figure I would give some updates for some of the gits just in case PyPi was easier. I have been trying to play around with virtualenvs too today but with little success. Still trying to wrap my head around Salt and all. And I keep finding more things to throw the devs way lol. I would love to see Salt support Pipx one day so that they could all get their own virtualenv for each tool and their dependencies. Hopefully one day soon that will come.

Cheers!

[geeksailor]

Not sure if the dev's have someone familiar with both Salt and Go but the integration does not seem easy from what I'm finding so far. I think I found a package that can tie them together to be able to run Go commands in Salt to build the package. It requires Ruby and Docker to run it but it looks like those are already install requirements for the current Remnux package so nothing super extra there. https://github.com/saltstack-formulas/golang-formula/. I am trying to install and mess with it now to see if I can make it work. If I figure it out I'll upload my salt. The old Docker-Compose Python route may be more feasible for now till a full solution can be found to make it work or decision on whether or not to just stick with the v1 of the program.

[geeksailor]

Ok so it has been a pain so far but here is what I got for this golang package I linked above:

• Git cloned the package onto my system and ran the commands as it says in the README to install it, however ran into a error on install that required troubleshooting to see if I could fix it. -- Troubleshooting the issue I ran into trying to do this install process, it turns out that this salt-go package or Kitchen or something uses Pigz which uses Zlib to uncompress some required files. There was a big bug in like v2.6 of Pigz that causes Pigz not to recognize the Zlib version, even if you have the most current version of Zlib. -- Another issue is that Pigz does not update its repo on Ubuntu for older OS versions to get its more current versions. So 20.04 can only pull 2.4, 22.04 can only get 2.6, and the most current version being 2.8. -- The Zlib version issue in Pigz has been patched as of August, but the newer version with this fix can't be pulled via repo. -- The fix I found is to either pull Pigz source from their git https://github.com/madler/pigz or their site http://zlib.net/pigz/ and compile it (a simple make command) to make the most current version. Then move the newly compiled programs (pigz & unpigz) into the /usr/bin to override the current version installed (if it is already installed). This will get the system to recognize that 2.8 is installed.
• Once the newly compiled files are dropped in, it should work no problem to run the kitchen test fully
• After this I am stuck since I have no experience with Salt and building pillars so trying to figure out how to integrate the commands from this new integration to allow for go states is above my head.

It seems like it may be a complex path to try to build a Salt to make this to work with the Remnux package just to get a newer version of Docker Compose to work with it. Again not sure if just sticking with the old v1 in Python may be the best bet just to lessen complications. But I'm stuck at this point and will leave it to someone else to see if they can find a simpler way or if they continue forward, a way to Salt all of this.

[digitalsleuth]

The golang build shouldn't be an issue at all, it's more along the lines of how compose is used and the support for various versions of the compose files. There is a compose binary which can simply be installed and bypass these issues.

[geeksailor]

Ah, glad you have a simpler solution than what I tried to figure out. I'm sure you got this. Cheers!

[digitalsleuth]

Just an update (haven't forgotten about this):

• I've created virtualenvs for qiling, speakeasy, and thug and have finished most of the testing. The last bit of testing I'll need to do is ensure the old versions of these tools are completely removed before installing the new virtualenv versions. This will likely be done tomorrow sometime.
• I've adapted docker-compose to the latest version, but it doesn't use the new V2 plugin, it still uses the old command. However, it does still function as the V2 plugin should.
• I haven't finished reviewing all of the git install methods via pip, but I've flagged a few which will need to remain. More will follow in the next day or two.

[geeksailor]

@digitalsleuth Hey buddy, It's all good. I know we all got life and other things going on and you would get to these issues when you could. But I am glad to see you managed to find some fixes through everything else you may currently have going on. I'm on vacation for a while longer but I'll do a pull for updates to Remnux once I get back to see if there is any issues that I can find. Thank you for the update and Happy New Years! Cheers!

[digitalsleuth]

@geeksailor Enjoy the rest of your vacation, and thanks and Happy New Years to you too!

[digitalsleuth]

@geeksailor With recent releases, you should find that the questions you asked above should now all be answered. Please let us know if you have any other concerns, and if this issue can be closed.

Cheers!

[geeksailor]

@digitalsleuth Sounds good! I'll close this thread out. I am back at work this week so I'm pushing updates to my system today and then I'll see if find any new issues after that (which I hopefully I shouldn't). If I do find anything though I'll start up a new thread. Thank you again for all your work to fix these issues. Cheers!