Setup Script Rewrite

[0x6d6f7468]

Intro

Closes #639. This is the rewritten setup.sh script for the Python version of EyeWitness.

Note that this does not close issue 636. I had naively put in a hack fix to force pip to break system packages but after discussion on that issue, I decided it needed more R&D to determine the best solution.

Changes

Notable changes/features are as follows:

1. Reduced code redundancy
2. Refactored Python dependencies to a requirements.txt file
3. Cleaned up global variables
4. Adjusted how OS version is determined and matched for
5. Cleaned up and improved geckodriver retrieval (closes #635)
   1. Removed static geckodriver version in lieu of a call to the GitHub API to get the latest version of geckodriver (closes #601)
      • In case this one is controversial: looking back at the geckodriver related issues shows a clear trend of "upgrading geckodriver" being the fix. to the reported issue.
      • I don't see a reason to anchor a static version that needs to be updated, but I'm happy to hear arguments to the contrary.
   2. Bolstered logic for CPU architecture checks for geckodriver. Adds aarch64 support (closes #577)
6. Other slight quality-of-life tweaks

Testing

Testing has been performed and the setup is confirmed to work on the following operating systems:

• Debian 12
• Ubuntu 22.04.3
• Linux Mint
• Parrot Security
• Kali Rolling
• CentOS Stream 9
• Rocky 9

Testing was performed on Fedora 39 but encountered errors that couldn't be bypassed with the simple PEP 668 hack (that I removed from this PR, more research is needed).

I have not thoroughly tested on:

• CentOS 7
  • I was unable to get an installation ISO to launch, but I think it will work if it worked on Stream 9.
• Alpine
  • I tried to get an Alpine VM set up but I was unsuccessful.
  • That said, I didn't end up changing anything in the dependency installation for Alpine from the original setup.sh. If I manage to get a good setup going, I might revisit that in the future to see if we can't use a newer version of Firefox, among other things.
• Arch/Manjaro
  • I have not yet been able to get a good Arch or Manjaro image on the aarch64 system I'm working with.
  • That said, yada yada same boat as Alpine.

If anybody would like to try and test the setup script on a CentOS 7, Alpine, Arch, or Manjaro machine, please let me know if you run into any snags.

Please let me know if you have any remarks on this. It's a pretty big change and I certainly don't want to undo a lot of existing work without a good chance for people to give feedback.

Thanks!

[digininja]

An extra that would be really useful for this would be the ability to pass in a proxy for it to use for all the actions.

I have a client that I run an internal test for every month or so and the box they give me doesn't have web access so I have to ssh tunnel squid in and then setup a load of configs so that when I run the setup script all the tools go through the proxy.

apt, wget and pip all support the environment variables for suggesting a proxy so this works:

export http_proxy=http://localhost:3128 export https_proxy=http://localhost:3128

But, if I do this for the shell, then when I run EW it tries to use the proxy as well and blows up trying to hit the internal URLs.

[0x6d6f7468]

Hey Robin, thanks for the feedback. I suppose I could add a config section to the setup script or somehow add command line arguments, but would it work to just run the script with sudo http_proxy=http://localhost:3128 https_proxy=http://localhost:3128 bash setup.sh? You can assign variables inline after the sudo and they should persist throughout the duration of the shell script execution.

[digininja]

I've not tried that, mostly because I have to setup apt and wget to work full time with proxies so I just edit their config files (root's wgetrc) rather than setting the variables.

On Mon, 22 Jan 2024 at 16:57, moth @.***> wrote:

Hey Robin, thanks for the feedback. I suppose I could add a config section to the setup script or somehow add command line arguments, but would it work to just run the script with sudo http_proxy=http://localhost:3128 https_proxy=http://localhost:3128 bash setup.sh? You can assign variables inline after the sudo and they should persist throughout the duration of the shell script execution.

— Reply to this email directly, view it on GitHub https://github.com/RedSiege/EyeWitness/pull/640#issuecomment-1904420375, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA4SWMN6UCT4QCKBVGIXOLYP2LA5AVCNFSM6AAAAABCBERGIWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBUGQZDAMZXGU . You are receiving this because you commented.Message ID: @.***>

[0x6d6f7468]

Ok. Do you see my suggestion as being sufficient for your suggestion at least? Or are there caveats that you can think of that I'm missing? Just want to make sure you're satisfied with it. I would love to avoid making people modify the setup script or a config somewhere if another solution exists, but that's just me. Thanks.

[digininja]

I get my next fresh box next Monday so I'll give this a try then and see.

On Mon, 22 Jan 2024, 17:09 moth, @.***> wrote:

Ok. Do you see my suggestion as being sufficient for your suggestion at least? Or are there caveats that you can think of that I'm missing? Just want to make sure you're satisfied with it. I would love to avoid making people modify the setup script or a config somewhere if another solution exists, but that's just me. Thanks.

— Reply to this email directly, view it on GitHub https://github.com/RedSiege/EyeWitness/pull/640#issuecomment-1904441452, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA4SWIPXJZFQIGXN32ICITYP2MLLAVCNFSM6AAAAABCBERGIWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBUGQ2DCNBVGI . You are receiving this because you commented.Message ID: @.***>

[digininja]

I just tried the new setup script through a proxy. It looks like the environment variable setting before running the script works, but apt through a proxy is so slow that it is hard to tell for the first couple of minutes if it is working or not. This is not your problem, just apt being slow.

The script nearly worked, towards the end I got this:

[*] Getting latest Gecko driver...
http://: Invalid host name.
tar: This does not look like a tar archive

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now

[+] Setup script completed successfully. Enjoy EyeWitness! ^_^
[*] Be sure to check out Red Siege!
[*] https://www.redsiege.com

Looking at the script, this is what failed:

└─$ https_proxy=http://localhost:3128  curl -s https://api.github.com/repos/mozilla/geckodriver/releases/latest | grep browser_download_url | cut -d':' -f2,3 | tr -d \" | tr -d '[:blank:]'
https://api.github.com/repos/mozilla/geckodriver/releases/135651589,assets_url

When you then try to find the 64-bit version, there is nothing to find and so you get a broken URL.

[0x6d6f7468]

Thanks for testing. I had seen that geckodriver bug but thought it was related to something else that I had solved.

If it's consistent behavior, could you possibly add the -L flag to the Curl command in the setup script to see if that fixes anything? It seems like certain conditions (such as using a proxy) cause a redirect to a different API endpoint but I'm not sure if it's due to an HTTP 301/302 or something else.

[digininja]

I've just ran this a load of times and ended up getting mixed results, it has worked a couple of times, but look at this from when it didn't work:

┌──(bstg㉿bstg-eee6-setup)-[~/client/nmap]
└─$ https_proxy=http://localhost:3128  curl -s -L https://api.github.com/repos/mozilla/geckodriver/releases/latest
{"url":"https://api.github.com/repos/mozilla/geckodriver/releases/135651589","assets_url":"https://api.github.com/repos/mozilla/geckodriver/releases/135651589/assets","upload_url":"https://uploads.github.com/repos/mozilla/geckodriver/releases/135651589/assets{?name,label}","html_url":"https://github.com/mozilla/geckodriver/releases/tag/v0.34.0","id":135651589,"author":{"login":"whimboo","id":129603,"node_id":"MDQ6VXNlcjEyOTYwMw==","avatar_url":"https://avatars.githubusercontent.com/u/129603?v=4","gravatar_id":"","url":"https://api.github.com/users/whimboo","html_url":"https://github.com/whimboo","followers_url":"https://api.github.com/users/whimboo/followers","following_url":"https://api.github.com/users/whimboo/following{/other_user}","gists_url":"https://api.github.com/users/whimboo/gists{/gist_id}","starred_url":"https://api.github.com/users/whimboo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/whimboo/subscriptions","organizations_url":"https://api.github.com/users/whimboo/orgs","repos_url":"https://api.github.com/users/whimboo/repos","events_url":"https://api.github.com/users/whimboo/events{/privacy}","received_events_url":"https://api.github.com/users/whimboo/received_events","type":"User","site_admin":false},"node_id":"RE_kwDOAYLgmc4IFeEF","tag_name":"v0.34.0","target_commitish":"release","name":"0.34.0","draft":false,"prerelease":false,"created_at":"2024-01-03T09:55:26Z","published_at":"2024-01-03T09:42:44Z","assets":[{"url":"https://api.github.com/repos/mozilla/geckodriver/releases/assets/143548693","id":143548693,"node_id":"RA_kwDOAYLgmc4IjmEV","name":"geckodriver-v0.34.0-linux-aarch64.tar.gz","label":null,"uploader":{"login":"whimboo","id":129603,"node_id":"MDQ6VXNlcjEyOTYwMw==","avatar_url":"https://avatars.githubusercontent.com/u/129603?v=4","gravatar_id":"","url":"https://api.github.com/users/whimboo","html_url":"https://github.com/whimboo","followers_url":"https://api.github.com/users/whimboo/followers","following_url":"https://api.github.com/users/whimboo/following{/other_u

That is the same JSON as I got when it worked, but without the newlines on it, that is why your command is failing. The grep gets the whole response because it contains the phrase, the cut then takes the first URL it sees because that is field 2 and 3.

So it isn't a redirect thing, it is something, probably at Github, that is messing up the returned string. I think you could probably handle it by saving the download then changing the way you grep for stuff so it works in either situation.

[0x6d6f7468]

Yeah I think you're right. It's so strange. Thanks for giving that a try, I was having trouble replicating it for testing which makes troubleshooting much more difficult. I'll let you know when I come up with something different.

[0x6d6f7468]

Half-tempted to just add jq to the dependency list and see if we can't get away with traditional JSON blob parsing...

Love me some jq.

[digininja]

A quick way to replicate it would be to grab the page, strip all the newlines, and put it on your own server. Or just stick it in a file and cat it into whatever pipeline you are writing.

On Tue, 30 Jan 2024, 20:38 moth, @.***> wrote:

Yeah I think you're right. It's so strange. Thanks for giving that a try, I was having trouble replicating it for testing which makes troubleshooting much more difficult. I'll let you know when I come up with something different.

— Reply to this email directly, view it on GitHub https://github.com/RedSiege/EyeWitness/pull/640#issuecomment-1917853551, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA4SWNVEY6Z37C4KPKYS5TYRFK53AVCNFSM6AAAAABCBERGIWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJXHA2TGNJVGE . You are receiving this because you commented.Message ID: @.***>

[0x6d6f7468]

Do you think it'd be acceptable to add jq to dependency list and then just do parsing with jq '.assets[].browser_download_url'? That appears to work with JSON blobs of either format, and requires fewer different parsing steps. Parsing would then look like this:

curl -s https://api.github.com/repos/mozilla/geckodriver/releases/latest | jq '.assets[].browser_download_url' | tr -d \"

The switch case can then remain the same.

[digininja]

You shouldn't as you don't actually need it to run the app so you leave stuff on the box that doesn't need to be there.

But I'd do it and if anyone complains tell them to submit a PR.

On Wed, 31 Jan 2024, 16:45 moth, @.***> wrote:

Do you think it'd be acceptable to add jq to dependency list and then just do parsing with jq '.assets[].browser_download_url'? That appears to work with JSON blobs of either format, and requires fewer different parsing steps.

— Reply to this email directly, view it on GitHub https://github.com/RedSiege/EyeWitness/pull/640#issuecomment-1919495323, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA4SWLZX3V74BKAJFNMX3LYRJYMBAVCNFSM6AAAAABCBERGIWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJZGQ4TKMZSGM . You are receiving this because you commented.Message ID: @.***>

[0x6d6f7468]

Fair point, though there are already a small handful of commands that we take for granted as installed but are now in the dependency list because they're not guaranteed.

Perhaps your suggestion is good though, we can just let people complain about dependencies if they want to in the future.

[digininja]

Just do it!

On Wed, 31 Jan 2024, 17:05 moth, @.***> wrote:

Fair point, though there are already a small handful of commands that we take for granted as installed but are now in the dependency list because they're not guaranteed.

Perhaps your suggestion is good though, we can just let people complain about dependencies if they want to in the future.

— Reply to this email directly, view it on GitHub https://github.com/RedSiege/EyeWitness/pull/640#issuecomment-1919533502, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA4SWIIEPJHZHDESKIO2YDYRJ2ULAVCNFSM6AAAAABCBERGIWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJZGUZTGNJQGI . You are receiving this because you commented.Message ID: @.***>

[0x6d6f7468]

Alright, let's give this a try. I'll need to test this on a RHEL-flavored VM as well just to make sure the jq package installs nicely. If you have a chance to test this, let me know if it still causes issues.

Thanks.

[digininja]

I've just tried it on a box that already has EW installed and the script worked fine, no errors and the geckodriver installed correctly.

It is a couple of weeks I think till I get a new blank box, I'll test it on there when it arrives.

On Wed, 31 Jan 2024 at 17:52, moth @.***> wrote:

Alright, let's give this a try. I'll need to test this on a RHEL-flavored VM as well just to make sure the jq package installs nicely. If you have a chance to test this, let me know if it still causes issues.

Thanks.

— Reply to this email directly, view it on GitHub https://github.com/RedSiege/EyeWitness/pull/640#issuecomment-1919614953, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA4SWK7SK55H4N26WAK2OTYRKAHBAVCNFSM6AAAAABCBERGIWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJZGYYTIOJVGM . You are receiving this because you commented.Message ID: @.***>

[Relkci]

We should probably get a list of distros we want this tested on.

Anyway, tested new setup script-- no issues

OS: Kali GNU/Linux Rolling x86_64 Kernel: 6.6.9-amd64

[mr-pmillz]

I think a nice long term support feature solution for automated testing that the setup script works properly would be to create GitHub Actions Workflow(s) that run the setup script on all the intended supported distros.

[Relkci]

Tested working on docker /w debian image.

will merge.