search

weidai11 / cryptopp

free C++ class library of cryptographic schemes
https://cryptopp.com
Other
4.89k stars 1.51k forks source link

Continuous integration and testing #97

Closed noloader closed 7 years ago

noloader commented 8 years ago

Continuous integration and testing is a topic that comes up frequently. Crypto++ testing is believed to be thorough on common platforms like Linux, OS X and Windows, but its a manual process. Tools like Travis-CI can be a helpful to automate testing, but Crypto++ does not have a process in place to utilize it.

Travis-CI is often recommended, but it requires dangerous GitHub permissions on a GitHub hosted project. Also see Travis-CI Issue 1390: Travis should not need write access to GitHub profile, followers, emails, etc...

For Crypto++, it appears Travis-CI effectively requires GitHub Collaborator status. If the Travis-CI service is compromised, then it could have potentially damaging effects on Crypto++. From The Linux Backdoor Attempt of 2003 (server compromise and insertion of backdoors) and Target data breach (compromise a secondary vendor/partner to gain access to the primary data), we know attackers will use all means to compromise a code base, so the risk is non-trivial. In fact, in 2007, SourceForge.net got hacked which contaminated all projects SF hosted, including Crypto++.

We'd like to find a way to make Travis-CI work with GitHub without exposing the project to unneeded risk. Or, we would like to find a replacement service that can be used more safely. Please provide suggestions, comments and thoughts on:

Also see Issue 97: Continuous integration and testing on the Crypto++ mailing list.

noloader commented 8 years ago

There seems to be some other difficulties with Travis-CI. First, when I visit https://www.travic-ci.org and click _Sign Up_, it wants to use my personal GitHub account. Second, when I open a private browsing window (so my personal GitHub info is not available), it takes me to GitHub to sign up for an account. So it appears I cannot create a Travis-CI account and subsequently ask it to test Crypto++ under Wei's GitHub.

Travis also appears to place a limit on CPU cycles, which is understandable. The limit is reported as 50 minutes; however, our cryptest.sh script takes about 4 hours to run on a modern x86_64 platform.

IlyaBizyaev commented 8 years ago

Hello, Jeffrey! Travis-CI allows you to test only your own repositories. So, if you want to test Crypto++, you have to either fork it and run Travis or to log in using Wei Dai's account. If he activates Crypto++ testing once, everyone will be able to access testing results in all the future releases.

noloader commented 8 years ago

@IlyaBizyaev -

or to log in using Wei Dai's account

Wei is somewhat risk averse. I'm guessing he is going to shy away from allowing Travis-CI the permissions. (I'm also risk averse, and I don't feel comfortable with them).

I see Travis-CI is open source. I also see its available for CentOS. Maybe we can run an instance on the www.cryptopp.com server. Do you have any experience with running your own instance of the service?

noloader commented 8 years ago

What we can use in lieu of Travis-CI

We also came across Buildbot. It looks like we can set it up on cryptopp.com as a master, and then allow it to provide both the i686 and x86_64 slaves. I can provision 3 or 4 ARM devices locally as slaves to the cryptopp.com master.

noloader commented 8 years ago

I'm still working on this... The Buildbot tutorial and 5 minute first run (see here) does not work on two different machines. The master does not start.

I need to take more time to figure out exactly what is broken behind the scenes. It appears to be something with Python and the Sandbox.

noloader commented 7 years ago

I've been running Travis Appveyor from my fork of Crypto++.

The downside is I have to manually sync with Wei's Crypto++. Git and hosts like GitHub ad Gist need to address that problem of manual sync's. What they have inadvertently done is taken the Maven Repo problem of one repo with old, outdated unpatched and insecure software and turned it into the Git Repo problem by exponentiating it which results in millions of forks that are old, outdated unpatched and insecure.