port Zinc-WebSocket packages to GemStone

[dalehenrich]

Paul (@pdebruic),

Just this week, Sebastian (@HeSe) has blown the dust off the Cryptography project[1] to support his port of Web-Sockets to GemStone[2].

Since my inclination is to move projects off of GemSource and onto GitHub when we start active work, I started poking around the current state of Cryptography, since the last time that project had been touched was 4 years ago.

On SqueakSource there is a Cryptography project[3] that appears to have a newer versions of the Cryptography and PasswordHashingFFI packages. this project has a number of interestingly named packages that may or may not be useful to have ported to GemStone. The Blowfish package is interesting because there's a pretty recent version not to mention the fact that you've got a blowfish project up on github[4].

In the pharo universe it seems that there is some recent activity with ApplicationSecurity[5] and Nacl[6] projects.

I am a little baffled by the lay of the land as there seem to be quite a variety of apis available and I'm not sure which ones apply to which problems ...

Soo ... I am curious about your thoughts on these various projects and the idea of consolidating the current work on GsDevKit ... part of the trick here is to define the "current work" and also look at where we "should be" ... Of course, I want to support existing packages that are used in existing projects and I'd like to start addressing performance issues ...

I would like to support packages that are available on Pharo and Squeak and get them ported to GemStone... If GemStone needs to add fundamental capabilities at the image level then now is a good time as we start work on the 3.3 release ...

Dale

[1] http://seaside.gemtalksystems.com/ss/Cryptography.html [2] https://github.com/GsDevKit/zinc/issues/58 [3] http://www.squeaksource.com/Cryptography.html [4] https://github.com/pdebruic/blowfish-smalltalk [5] http://www.smalltalkhub.com/mc/hernan/ApplicationSecurity/main [6] http://www.smalltalkhub.com/mc/tonyg/Crypto-Nacl/main

[jbrichau]

FYI: In the summer of 2013, I (urgently) needed an implementation of SHA256 in GemStone. I tried to blow the dust of the Cryptography package for GS but I had to limit myself to the one or two packages (or parts thereof) of the project I needed to run SHA256.

So, not all is in ruins but I did notice that a lot of the things were not working. Also in Pharo, the project does not load anymore as various parts have been copy/pasted into the image.

[HeSe]

Hi Johan,

Well, I just had a look. It seems the hash functions are still doing fine. Many errors are due to a different hex implementation in both Gemstone portability layers and Squeak/Pharo

Whereas in gemstone hex is delivered in lowercase it is uppercase in Pharo. The values are right though. We might need to clarify if there is a need for Gemstone to have hex in class ByteArray handling lowercase representation of a hex.

Would it make sense to at least do obvious changes and add Cryptography as prerequisite to Zinc-Websockets for Gemstone?

Sebastian

On 2014-11-02 7:36 AM, Johan Brichau wrote:

FYI: In the summer of 2013, I (urgently) needed an implementation of SHA256 in GemStone. I tried to blow the dust of the Cryptography package for GS but I had to limit myself to the one or two packages (or parts thereof) of the project I needed to run SHA256.

So, not all is in ruins but I did notice that a lot of the things were not working. Also in Pharo, the project does not load anymore as various parts have been copy/pasted into the image.

— Reply to this email directly or view it on GitHub https://github.com/GsDevKit/zinc/issues/58#issuecomment-61411163.

[pdebruic]

Hi Dale,

We should probably hop on the cryptography list: http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/cryptography I've cc'd it for now.

I still use the PasswordHashingFFI code on squeak/pharo and GemStone. Its pretty basic, but useful. With libxcrypt installed, & on linux, it lets a user hash passwords using bcrypt or SHA-512 (but the password hashing version). A primer on why you'd want to do that is here: http://codahale.com/how-to-safely-store-a-password/ . Another good, slow, password hashing algorithm is scrypt (https://en.wikipedia.org/wiki/Scrypt) but we don't have an FFI library for it yet. In addtiion to Hernan's Application Security project Pierce Ng has some projects he mentions on his blog: http://www.samadhiweb.com/tags/cryptography

Blowfish is a 2 way encryption algorithm which allows someone to encrypt something and then decrypt it later. The version I wrote only encrypts an 8 byte block at a time. Anything longer looks encrypted but isn't. To do longer things someone would need to add code that does cipher block chaining (CBC) (e.g. https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher-block_chaining_.28CBC.29) I never added the CBC code because I wrote the Blowfish code while investigating having a Smalltalk bcrypt password hashing implementation. Bcrypt uses blowfish while hashing passwords for long term storage. The Smalltalk implementation ended up being more than 5,000x slower than the FFI version and not usable in practice. From my point of view right now the Blowfish code shouldn't be used for anything without adding the CBC code and even then a person may be better off using a newer algorithm than Blowfish for 2 way encryption because of both hardware and algorithm advances (e.g. the NaCl library you mention). That being said the Smalltalk Blowfish implementation does work for 8 byte blocks and is only Smalltalk, no FFI.

I think its a good idea for some tidying in the Cryptography ecosystem, but don't have a strong idea of what the end state (packages and their location on the internet) should be.

Hope this helps

Paul

On Nov 1, 2014, at 11:20 AM, Dale Henrichs notifications@github.com wrote:

Paul (@pdebruic),

Just this week, Sebastian (@HeSe) has blown the dust off the Cryptography project[1] to support his port of Web-Sockets to GemStone[2].

Since my inclination is to move projects off of GemSource and onto GitHub when we start active work, I started poking around the current state of Cryptography, since the last time that project had been touched was 4 years ago.

On SqueakSource there is a Cryptography project[3] that appears to have a newer versions of the Cryptography and PasswordHashingFFI packages. this project has a number of interestingly named packages that may or may not be useful to have ported to GemStone. The Blowfish package is interesting because there's a pretty recent version not to mention the fact that you've got a blowfish project up on github[4].

In the pharo universe it seems that there is some recent activity with ApplicationSecurity[5] and Nacl[6] projects.

I am a little baffled by the lay of the land as there seem to be quite a variety of apis available and I'm not sure which ones apply to which problems ...

Soo ... I am curious about your thoughts on these various projects and the idea of consolidating the current work on GsDevKit ... part of the trick here is to define the "current work" and also look at where we "should be" ... Of course, I want to support existing packages that are used in existing projects and I'd like to start addressing performance issues ...

I would like to support packages that are available on Pharo and Squeak and get them ported to GemStone... If GemStone needs to add fundamental capabilities at the image level then now is a good time as we start work on the 3.3 release ...

Dale

[1] http://seaside.gemtalksystems.com/ss/Cryptography.html [2] #58 [3] http://www.squeaksource.com/Cryptography.html [4] https://github.com/pdebruic/blowfish-smalltalk [5] http://www.smalltalkhub.com/mc/hernan/ApplicationSecurity/main [6] http://www.smalltalkhub.com/mc/tonyg/Crypto-Nacl/main

— Reply to this email directly or view it on GitHub.

[dalehenrich]

Thanks @pdebruic, @HeSe and @jbrichau ... I've subscribed to the cryptography list so that I can follow any responses there, but I I am inclined to think that we in the GemStone community are the only ones with a vested interest in a cross-platform cryptography library so it seems that for us the best solution would be to create a Cryptography project on GsDevKit and try to maintain the subset of the library that is being actively used by the GemStone community (with tests on Pharo and Squeak) and expand the subset as needed ... we should try to mirror the new packages on the SqueakSource list as we do with zinc, seaside, etc.

If the discussion on the cryptography list heats up, I'm inclined to follow their liead ...

What do you guys think?

[jbrichau]

Sound like a plan. I’m going to have to focus on Seaside and XMLParser for now, but I can definitely already check out the status of the packages of Cryptography that we are using.

Johan

On 03 Nov 2014, at 19:07, Dale Henrichs notifications@github.com wrote:

Thanks @pdebruic https://github.com/pdebruic, @HeSe https://github.com/HeSe and @jbrichau https://github.com/jbrichau ... I've subscribed to the cryptography list so that I can follow any responses there, but I I am inclined to think that we in the GemStone community are the only ones with a vested interest in a cross-platform cryptography library so it seems that for us the best solution would be to create a Cryptography project on GsDevKit and try to maintain the subset of the library that is being actively used by the GemStone community (with tests on Pharo and Squeak) and expand the subset as needed ... we should try to mirror the new packages on the SqueakSource list as we do with zinc, seaside, etc.

If the discussion on the cryptography list heats up, I'm inclined to follow their liead ...

What do you guys think?

— Reply to this email directly or view it on GitHub https://github.com/GsDevKit/zinc/issues/58#issuecomment-61520755.

[dalehenrich]

For starters we'll be referencing the new Cryptography project from Zinc on the issue_58 branch and then only from the Web-Socket package ... get all the tests green and then move from there ...

On Mon, Nov 3, 2014 at 12:41 PM, Johan Brichau notifications@github.com wrote:

Sound like a plan. I’m going to have to focus on Seaside and XMLParser for now, but I can definitely already check out the status of the packages of Cryptography that we are using.

Johan

On 03 Nov 2014, at 19:07, Dale Henrichs notifications@github.com wrote:

Thanks @pdebruic https://github.com/pdebruic, @HeSe < https://github.com/HeSe> and @jbrichau https://github.com/jbrichau ... I've subscribed to the cryptography list so that I can follow any responses there, but I I am inclined to think that we in the GemStone community are the only ones with a vested interest in a cross-platform cryptography library so it seems that for us the best solution would be to create a Cryptography project on GsDevKit and try to maintain the subset of the library that is being actively used by the GemStone community (with tests on Pharo and Squeak) and expand the subset as needed ... we should try to mirror the new packages on the SqueakSource list as we do with zinc, seaside, etc.

If the discussion on the cryptography list heats up, I'm inclined to follow their liead ...

What do you guys think?

— Reply to this email directly or view it on GitHub < https://github.com/GsDevKit/zinc/issues/58#issuecomment-61520755>.

— Reply to this email directly or view it on GitHub https://github.com/GsDevKit/zinc/issues/58#issuecomment-61544827.

[dalehenrich]

@pdebruic, @HeSe and @jbrichau ... Cryptography project added for now it is just serving the two packages that have been ported to GemStone, slthough I did copy all of the packages from the SqueakSource project for good measure. Didn't add them to baseline because at this moment don't know which ones run in Squeak or Pharo or nowhere ...

On the issue_58 branch I've updated the baseline to reference the GsDevKit/Cryptography project, but we have a number of test failures ...

Hooked up the NeoJSON project as well, and it has some failing tests, too ...

I'm setting up the Cryptography project on travis so I'll be looking into getting things to pass there ...

[dalehenrich]

@pdebruic, @HeSe and @jbrichau ... Cryptography tests are green across the range of GemStone version

[dalehenrich]

Switched to running all unit tests in image to pick up errors that may be introduced (like SentButNotImplemented and UndefinedSymbols) and that may be failing from required projects and here's the tale of tape... mainly websocket test failures:

**************************************************************************************
    Results for TestCase
1061 run, 1041 passes, 3 expected defects, 2 failures, 15 errors, 0 unexpected passes
**************************************************************************************
*** FAILURES *******************
    SentButNotImplementedTest debug: #'testSentButNotImplemented'.
    UndefinedSymbolsTest debug: #'testUndefinedSymbols'.
*** ERRORS *******************
    ZnWebSocketFrameTests debug: #'testReading256ByteSingleFrameUnmaskedBinaryMessage'.
    ZnWebSocketFrameTests debug: #'testReading256KiBSingleFrameUnmaskedBinaryMessage'.
    ZnWebSocketFrameTests debug: #'testReadingFragmentedUnmaskedTextMessage'.
    ZnWebSocketFrameTests debug: #'testReadingPing'.
    ZnWebSocketFrameTests debug: #'testReadingPong'.
    ZnWebSocketFrameTests debug: #'testReadingSingleFrameMaskedTextMessage'.
    ZnWebSocketFrameTests debug: #'testReadingSingleFrameUnmaskedTextMessage'.
    ZnWebSocketFrameTests debug: #'testWriting256ByteSingleFrameUnmaskedBinaryMessage'.
    ZnWebSocketFrameTests debug: #'testWriting256KiBSingleFrameUnmaskedBinaryMessage'.
    ZnWebSocketFrameTests debug: #'testWritingPing'.
    ZnWebSocketFrameTests debug: #'testWritingSingleFrameUnmaskedTextMessage'.
    ZnWebSocketTests debug: #'testChatroom'.
    ZnWebSocketTests debug: #'testEcho'.
    ZnWebSocketTests debug: #'testEchoSecureWebSocketsDotOrg'.
    ZnWebSocketTests debug: #'testStatus'.
**************************************************************************************

[dalehenrich]

slug of sentButNotImplemented messages (not sure where they are all coming from)

[dalehenrich]

Note that the class ZnServerStarter is needed for the ZnWebSocketTests and that class will need a custom bash script for starting a separate server for the tests (thanks @HeSe ). So we may have to wait until I change buiderCI to use gsDevKitHome for installing GLASS/GsDevKit before getting the tests to run clean...

[dalehenrich]

It looks like some of the websocket code depends upon some methods that are in STON ... so will need to migrate those methods into GsSqueakCommon or GsPharo ....

[dalehenrich]

The class ZNServerStarter is over in https://github.com/HeSe/gsApplicationTools

[dalehenrich]

clean sheet for Cryptography tests

[dalehenrich]

clean sheet for gsApplicationTools tests ... the gsApplicationTools project provides pretty generic topaz/gemserver launch capabilities ... I plan to use gsApplicationTools for seaside gems, maintenance vm and service vm ....

[dalehenrich]

With this most recent commit, a continuation is snapped off within the zinc infrastructure when an error occurs ... I don't think that this will interfere with the seaside handling of errors, since this will happen only when errors make there way out the zinc universe ... but it is something to be aware of ... It's probably a good idea to see if I can make this a bit more configurable ...

[dalehenrich]

Looks like ZnLogSupport can be swizzled on a server by server basis, so I could arrange to have a subclass of ZnLogSupport that will allow for the customization of the class of ZnLogEvent used which will allow for the substitution of continuation creation variant for non-seaside server applications ...

[dalehenrich]

Looks like it might be interesting to have a continuationMode similar to debugMode to control continuations ...

[dalehenrich]

And another thing ... besides error handling ... should have Breakpoint and Halt handlers that snap off continuation then resume ... if in continuation mode is true ... trying to get to the point where GemServer can be developed remotely ... Breakpoints and Halt are critical ...

[dalehenrich]

clean sheet for 3.x webserver tests ... oh yeah, baby!

[dalehenrich]

Added Halt and Breakpoint handling ...here's a listing from ol view for a Chatroom zinc web socket handler with a self halt in a loop for the handler:

info        -->>Start ZnWebSocketTestChatServer on 1701            5335   12/03/2014 14:52:16:613
info        recordGemPid: ZnWebSocketTestChatServer on 1701        5335   12/03/2014 14:52:16:614
info        setStatmonCacheName: ZnWebSocketTestChatServer         5335   12/03/2014 14:52:16:615
info        enableRemoteBreakpointHandling: ZnWebSocketTestCha...  5335   12/03/2014 14:52:16:616
info        startSigAbortHandling: ZnWebSocketTestChatServer       5335   12/03/2014 14:52:16:671
info        299980289 read message...                              5335   12/03/2014 14:52:19:434
debug       Executing request/response loop                        5335   12/03/2014 14:52:19:485
info        Read aZnRequest(GET /ws-chatroom)                      5335   12/03/2014 14:52:19:535
trace       GET /ws-chatroom 101 1ms                               5335   12/03/2014 14:52:19:585
info        Wrote aZnWebSocketResponse(101 Switching Protocols...  5335   12/03/2014 14:52:19:643
info        300015873 registering                                  5335   12/03/2014 14:52:19:701
info        300015873 read message...                              5335   12/03/2014 14:52:19:802
info        300015873 got frame..                                  5335   12/03/2014 14:52:19:860
info        300015873 Received message: 'Message 2 @ 12/03/201...  5335   12/03/2014 14:52:19:919
transcript  'Creating continuation for: ''a Halt occurred (err...  5335   12/03/2014 14:52:19:919
error       -- continuation -- (a Halt occurred (error 2709))      5335   12/03/2014 14:52:19:920
info        299980289 Sending message: 'Message 2 @ 12/03/2014...  5335   12/03/2014 14:52:19:920
info        299980289 sent message                                 5335   12/03/2014 14:52:19:921
transcript  'Creating continuation for: ''a Halt occurred (err...  5335   12/03/2014 14:52:19:921
error       -- continuation -- (a Halt occurred (error 2709))      5335   12/03/2014 14:52:19:921
info        300015873 Sending message: 'Message 2 @ 12/03/2014...  5335   12/03/2014 14:52:19:922
info        300015873 sent message                                 5335   12/03/2014 14:52:19:922
info        300015873 distributed message                          5335   12/03/2014 14:52:19:922
info        300015873 read message...                              5335   12/03/2014 14:52:19:977
info        299980289 got frame..                                  5335   12/03/2014 14:52:20:127
info        299980289 Received message: 'Message 1 @ 12/03/201...  5335   12/03/2014 14:52:20:186
transcript  'Creating continuation for: ''a Halt occurred (err...  5335   12/03/2014 14:52:20:186
error       -- continuation -- (a Halt occurred (error 2709))      5335   12/03/2014 14:52:20:186
info        299980289 Sending message: 'Message 1 @ 12/03/2014...  5335   12/03/2014 14:52:20:187
info        299980289 sent message                                 5335   12/03/2014 14:52:20:187
transcript  'Creating continuation for: ''a Halt occurred (err...  5335   12/03/2014 14:52:20:187
error       -- continuation -- (a Halt occurred (error 2709))      5335   12/03/2014 14:52:20:188
info        300015873 Sending message: 'Message 1 @ 12/03/2014...  5335   12/03/2014 14:52:20:188
info        300015873 sent message                                 5335   12/03/2014 14:52:20:188
info        Stop Gems: ZnWebSocketTestChatServer                   29774  12/03/2014 14:52:20:189
info        299980289 distributed message                          5335   12/03/2014 14:52:20:189
info        performOnServer: ZnWebSocketTestChatServer             29774  12/03/2014 14:52:20:196

So similar to seaside you can set breakpoints of halts in code that'r running on a remote server and debug the halt and breakpoints from the object log...

[dalehenrich]

clean sheet for all platforms ... that means websocket tests are passing using a remote gem spawned using the gsApplicationTools script