search

openwall / john

John the Ripper jumbo - advanced offline password cracker, which supports hundreds of hash and cipher types, and runs on many operating systems, CPUs, GPUs, and even some FPGAs
https://www.openwall.com/john/
Other
10.14k stars 2.08k forks source link

Set up GitHub Actions #4543

Closed solardiz closed 9 months ago

solardiz commented 3 years ago

Some projects are moving from Travis CI to GitHub Actions, e.g. https://github.com/gost-engine/engine/pull/309/files

@claudioandre-br @magnumripper Should we?

claudioandre-br commented 3 years ago

IMO, it is interesting, especially, if someone plans:

But, for example, I didn't find any Ubuntu 14 in their virtual environments. So, there would be no more support for Linux OpenCL.

On the other hand, if you plan on having self-hosted runners, this would be an interesting approach. It's cheaper than Travis and you can run anything you want using your own hardware.

PS: Travis has a lot of useful base VMs. For a while, we need to avoid CI docker-based runners; the (Docker Hub) pull rate limit is now active.

solardiz commented 3 years ago

I didn't find any Ubuntu 14 in their virtual environments. So, there would be no more support for Linux OpenCL.

How is Ubuntu 14 related to OpenCL?

claudioandre-br commented 3 years ago

Citation:

For Ubuntu 16.04 LTS and above, the AMD Catalyst or fglrx driver is no longer supported by AMD

solardiz commented 3 years ago

For Ubuntu 16.04 LTS and above, the AMD Catalyst or fglrx driver is no longer supported by AMD

How is that relevant? First, on CPU it was AMD APP SDK, and not Catalyst nor fglrx. Also, why Ubuntu? APP SDK also worked e.g. on CentOS 7. (I don't know if they have that, but perhaps they do?)

Then, can't we use PoCL or/and Intel OpenCL?

solardiz commented 3 years ago

More examples:

https://github.com/strace/strace/blob/master/.github/workflows/ci.yml https://github.com/linux-pam/linux-pam/blob/master/.github/workflows/ci.yml https://github.com/altlinux/libutempter/blob/master/.github/workflows/ci.yml

claudioandre-br commented 3 years ago

I am not saying that JtR is prevented from doing something entirely different.

solardiz commented 3 years ago

As I understand, recent hashcat supports PoCL, so it's probably good enough for us to support, too.

By "slow", are you referring to Intel OpenCL taking a lot of time to build kernels? Is there possibly an optimization option we can tweak to favor lower build time over higher performance of the resulting code?

claudioandre-br commented 3 years ago

We support PoCL [1][2]. Unfortunately, not all formats work on it (warnings muted):

Device 1: pthread-AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx
Benchmarking: KeePass-opencl [SHA256 AES/Twofish/ChaCha OpenCL]...
[...]
Segmentation fault (core dumped)

./john --test --format=sha256crypt-opencl

Device 1: pthread-AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx Benchmarking: sha256crypt-opencl, crypt(3) $5$ (rounds=5000) [SHA256 OpenCL]... LWS=4 GWS=128 (32 blocks) DONE Speed for cost 1 (iteration count) of 5000 Raw: 1152 c/s real, 165 c/s virtual

./john --test --format=raw-sha512-opencl

Device 1: pthread-AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx Benchmarking: raw-SHA512-opencl [SHA512 OpenCL/mask accel]... LWS=8 GWS=2048 (256 blocks) x260 DONE Raw: 5945K c/s real, 801507 c/s virtual

./john --test --format=xsha512-opencl

Device 1: pthread-AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx Benchmarking: XSHA512-opencl, Mac OS X 10.7 salted [SHA512 OpenCL/mask accel]... LWS=8 GWS=1024 (128 blocks) x260 DONE Warning: "Many salts" test limited: 42/256 Many salts: 5481K c/s real, 738091 c/s virtual Only one salt: 5165K c/s real, 723579 c/s virtual


* Some CPU format values:

C:\Temp\JtR>run\john --test --format=sha512crypt Will run 8 OpenMP threads Benchmarking: sha512crypt, crypt(3) $6$ (rounds=5000) [SHA512 256/256 AVX2 4x]... (8xOMP) DONE Speed for cost 1 (iteration count) of 5000 Raw: 1753 c/s real, 238 c/s virtual

C:\Temp\JtR>run\john --test --format=sha256crypt Will run 8 OpenMP threads Benchmarking: sha256crypt, crypt(3) $5$ (rounds=5000) [SHA256 256/256 AVX2 8x]... (8xOMP) DONE Speed for cost 1 (iteration count) of 5000 Raw: 3255 c/s real, 423 c/s virtual


----------
**[EDITED]**

Intent/purpose: Intel OpenCL CPU Runtime integration into POCL

I forgot to announce this. It'sold news, but I'm sure people are not aware: https://sourceforge.net/p/pocl/mailman/pocl-devel/thread/1517615B361A5B4797D7811F9FD4F0FE22F23A3B%40IRSMSX107.ger.corp.intel.com/#msg36587306
solardiz commented 3 years ago

We support PoCL [1]. Unfortunately, not all formats work on it (warnings muted):

Device 1: pthread-AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx
Benchmarking: KeePass-opencl [SHA256 AES/Twofish/ChaCha OpenCL]...
[...]
Segmentation fault (core dumped)

This might be an opportunity for us to figure out and fix #4343.

As to the speeds vs. CPU formats, looks like PoCL doesn't vectorize, which is why sha512crypt-opencl and sha256crypt-opencl appear almost same speed. As long as vectorization is rarely beneficial on GPUs and we do also have CPU formats, we might not bother improving this, but we could use PoCL for testing.

solardiz commented 3 years ago

FWIW, we just got working GitHub CI for passwdqc today: https://github.com/openwall/passwdqc/pull/9

aktech commented 3 years ago

On the other hand, if you plan on having self-hosted runners, this would be an interesting approach.

Hi @claudioandre-br I am the creator of Cirun.io, "self-hosted runners" caught my eye.

FWIW I'll share my two cents. I created a service for problems like these, which is basically running custom machines (including GPUs) in GitHub Actions: https://cirun.io/

It is used in multiple open source projects needing GPU/custom machine support like the following:

It is fairly simple to setup, all you need is a cloud account (AWS or GCP) and a simple yaml file describing what kind of machines you need and Cirun will spin up ephemeral machines on your cloud for GitHub Actions to run. It's native to GitHub ecosystem, which mean you can see logs/trigger in the Github's interface itself, just like any Github Action run.

Also, note that Cirun is free for Open source projects. (You only pay to your cloud provider for machine usage)

claudioandre-br commented 3 years ago

Thanks @aktech.

I agree that it is a good option and would be economical for on-demand CI [1]. Anyway, since "we" already have on-premises hardware, I think a better approach would be to use self-hosted runners from GitHub Actions [2]. Also, I'm not sure we have funding to use a GPU on a cloud provider.

[1] Manual triggered:

on:
  workflow_dispatch

[2] https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners

solardiz commented 9 months ago

we just got working GitHub CI for passwdqc today: openwall/passwdqc#9

A newer revision of the above is now integrated in here via #5407. I think this is good enough to close this issue. There will always be room for improvement, such as adding OpenCL tests, but this issue is about the initial setup.