Replace raw-sha512-opencl with raw-sha512-ng-opencl again

solardiz commented 8 years ago

We have GPU mask mode for raw-sha256-opencl, but not for raw-sha512-opencl - perhaps we should correct this, especially given that it should be easy due to the similarity between these two.

claudioandre-br commented 8 years ago

As the best of my knowledge there is only one raw-sha256-opencl implementation.

That said, the twin brother of raw-sha256-opencl is raw-sha512-ng-opencl and it has GPU mask mode.

solardiz commented 8 years ago

Oh, you're right. IIRC, the issues with raw-sha512-ng-opencl preventing its rename (back) to just raw-sha512-opencl were poor performance on large number of loaded hashes, and license. Claudio, would you be willing to fix these two issues soon? Meanwhile, let's edit ALGORITHM_NAME in opencl_rawsha512_fmt_plug.c to suggest raw-sha512-ng-opencl as a possibly better alternative. Now it says "inefficient, development use mostly". Perhaps it should say "inefficient, please consider raw-sha512-ng-opencl". Unfortunately, this would give the impression that "ng" is always faster, which IIRC is not the case (not when a lot of hashes are loaded for cracking, where "ng" can be very slow). Please feel free to suggest better wording.

claudioandre-br commented 8 years ago

I tested it using (not sure) 5M hashes. Why poor performance on large number of loaded hashes?

claudioandre-br commented 8 years ago

Something like this (february):

raw-SHA512 [GPU]

Test	Candidates	Guesses	Status
[1]	-	-	593050K c/s
[2]	1	1	561819Kp/s
[3]	995	995	515681Kp/s
[4]	300,002	1	550342Kp/s
[5]	3,062,025	1	541935Kp/s

solardiz commented 8 years ago

Thanks, Claudio. I've just tested and can confirm that the slowness with many hashes loaded is gone. I had mentioned it on john-dev here http://www.openwall.com/lists/john-dev/2015/08/25/8 and it looks like you've fixed it since. Then it's just the license that's left to fix, and we can rename the format back from "ng" and move myrice's older one to "unused".

magnumripper commented 8 years ago

Surely the GPLv2 license alone can't be a blocker? Don't we have a lot of GPLv2 code in Jumbo?

claudioandre-br commented 8 years ago

I prefer the GPL license.

anyone can learn from and use the software.
and I love the viral nature of it.

solardiz commented 8 years ago

I am undecided on whether to consider it a blocker or not, but I certainly am not happy about continuing to "invest" in GPL'ed code in JtR. We do have more GPLv2+ code in jumbo, but I'd like us to decrease its amount. I plan on doing the same for core, likely relaxing the license for the entire core tree. With hashcat being MIT licensed, us insisting on GPL makes even less sense than it did before.

Maybe we can rename, but not move the other format to unused yet, because it does have this licensing advantage. What to call it, then? "free"? Of course, there are pros and cons to either license, including in terms of freedom, so that naming would be misleading.

Most immediately, this has relevance for LC7. I think unless Claudio relaxes the license for his formats at least to LGPL (with my preference being 0-clause BSD), or gives explicit permission for his code's reuse in jtrdll/LC7, we'll have to ask @crioux to exclude those formats from jtrdll. Maybe that's what Claudio actually wants (and ditto in potential future reuses like this), but I think that would actually be bad. We need to make a decision.

crioux commented 8 years ago

Claudio and I already discussed this, and he gave me permission to reuse the code in LC7.

crioux commented 8 years ago

That said, I'm all in favor of having that code move to non-gpl, in support of your efforts, Solar.

claudioandre-br commented 8 years ago

My understanding is: any tool at any time can use the kernel if it:

includes the source code, with or without modification; mandatory.
does the OpenCL kernel compilation at run-time; or use a GPL tool do create pre-compiled kernels (see previous item).

Once these criterias are meet, people can move on without my permission (GPL does permit).

Anyone else worried about the safety of the procedure, can ask for an extra 'permission'.

PS:

there is no linkage necessary. I see no problems with the spirit of GPL license.
if the source is available, anyone can even offer pre-compiled kernels (in case of complaints, create a GPL tool to do the compilation). In fact, it already exists.

solardiz commented 8 years ago

Claudio, these are all good points regarding OpenCL kernels. Makes sense to me, although I think a few of the things are subject to interpretation of the GPL, which varies. Some will argue that doing what you describe requires LGPL.

Yet it's so much cleaner to simply de-GPL the kernels, and there are also the C files. opencl_rawsha512_ng_fmt_plug.c currently has your copyright and is under GPLv2. We also need to check how Samuele's code that this file builds upon was licensed - but I vaguely recall it wasn't GPL'ed.

claudioandre-br commented 8 years ago

Thinking about the format file (*_fmt_plug.c), I guess an opensource library (loaded at runtime) would be necessary. Life is becoming harder, I see.

The main problem is that I see no reason to write non-GPL code.

Let's say an explicit permission is required and I granted one for LC7+.

claudioandre-br commented 8 years ago

To me the main question is: why are you Claudio writing software in your free time?

I write code to advertise it. Anyone can use it, even modify and sell it. In such a case, I would like to be able to say: look, I'm 'being used'.

GPL handles it. Easy-peasy.

magnumripper commented 8 years ago

So let's rename Myrice's format to raw-sha512-free-opencl and Claudio's to raw-sha512-opencl.

magnumripper commented 8 years ago

Don't we have a lot of GPLv2 code in Jumbo?

The below probably missed some:

$ git grep -i GPL *.c
BFEgg_fmt_plug.c: * This is a free software distributable under terms of the GNU GPL.
ike_fmt_plug.c: * and it is hereby released to the general public under GPL
john.c: * uses are under GPLv2.  For licensing terms for John the Ripper as a whole,
jumbo.c:   LGPL 2.1+ */
keepass2john.c: * is hereby released under GPL license.
opencl_autotune.c: * http://www.gnu.org/licenses/gpl-2.0.html
opencl_cryptsha256_fmt_plug.c: * http://www.gnu.org/licenses/gpl-2.0.html
opencl_cryptsha512_fmt_plug.c: * http://www.gnu.org/licenses/gpl-2.0.html
opencl_mysqlsha1_fmt_plug.c: * http://www.gnu.org/licenses/gpl-2.0.html
opencl_o5logon_fmt_plug.c: * and thus are licensed under the GPLv2.
opencl_rakp_fmt_plug.c: * Licensed under GPLv2
opencl_rakp_fmt_plug.c: * implied. See the following for more information on the GPLv2 license:
opencl_rakp_fmt_plug.c: * http://www.gnu.org/licenses/gpl-2.0.html
opencl_rawsha1_fmt_plug.c: * http://www.gnu.org/licenses/gpl-2.0.html
opencl_rawsha256_fmt_plug.c: * http://www.gnu.org/licenses/gpl-2.0.html
opencl_rawsha512_ng_fmt_plug.c: * http://www.gnu.org/licenses/gpl-2.0.html
opencl_salted_sha_fmt_plug.c: * http://www.gnu.org/licenses/gpl-2.0.html
opencl_sha2_common_plug.c: * http://www.gnu.org/licenses/gpl-2.0.html
pdf_fmt_plug.c: * Uses code from Sumatra PDF and MuPDF which are under GPL
pfx2john.c: * and it is hereby released under GPL.
twofish_plug.c: * under the GPL, include it in your commercial product without releasing
undrop.c: * This is a free software distributable under terms of the GNU GPL.
vnc_fmt_plug.c: * Licensed under the GNU GPL v2
vncpcap2john.c: * Licensed under the GNU GPL v2

Notable is opencl_autotune.c and opencl-autotune.h - nearly every OpenCL format use them.
twofish_plug.c that is a false positive - that one is free.

magnumripper commented 8 years ago

Picking opencl_rawsha1_fmt_plug.c as an example, I had a closer look at it. The GPL statement stems from Samuele's original code back in 2011. The current format and kernel has damn near nothing left from his code and certainly nothing significant. Yet the GPL tag still sits there. How would we go about getting rid of it?

magnumripper commented 8 years ago

Renamed in #2231

openwall / john

Replace raw-sha512-opencl with raw-sha512-ng-opencl again #2228