[discussion] Random number generator API

[igrr]

Arduino API has several functions exposing PRNG functionality: randomSeed, random(howbig), and random(howsmall, howbig). Normally these functions call software PRNG provided by libc.

Recently we have made a change (https://github.com/esp8266/Arduino/commit/bf067f718a29a4c9b2226d404bdcb4dcf926b5f8) which modified behaviour of these Arduino functions. These functions now mix the values from software PRNG and ESP8266 internal hardware RNG. This change raised a few questions on gitter which I'm moving here for better discussion.

1. Should we revert to the old behaviour to maintain compatibility? Assume that some sketch calls randomSeed function with the same seed each time to produce a repeatable pseudo-random sequence (like some games do). By altering the values with HW PRNG we are breaking compatibility for this application.
2. Some users will take random() and use it to generate nonces, keys, and other values which need better randomness than newlib's rand can provide. If we modify behaviour of random to use HW PRNG, we can improve security of these applications.
3. Yet another bunch of users (and some libraries) will do things like randomSeed(someFunc(analogRead(A0))). Clearly they are trying to get good random values, so probably the call to randomSeed function can not serve as a good indicator that user wants a SW PRNG behaviour.
4. Adding a new API like secureRandom() is one of the options. But we try to be compatible with Arduino libraries, and Arduino-compatible libraries will not be using secureRandom() unless it becomes part of the official Arduino API.
5. Adding configuration function like setPRNG(PRNG_SOFTWARE)/setPRNG(PRNG_HARDWARE) is yet another option. The trouble is, we still need to pick one of them as the default one. I.e. we need to choose between compatibility and security.

Please drop your thoughts/suggestions on this matter so that we can improve this feature for the next release.

/cc @skorokithakis @Makuna @Links2004 @holgerlembke who participated in the initial discussion on Gitter.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[Makuna]

I believe we need to maintain compatibility with the Arduino APIs. So having the default set for software is my belief.

I like the concept of having a "system.config" concept that a sketch can call that would switch the underlying random functions. This way libraries written to work across platforms will still function and now function with more security. Also instead of saying hardware, maybe use security/encryption so its obvious why you would use one over the other. PRNG_Compatibile, PRNG_Secure?

There will always be an issue where one library relied on compatibile and another would benefit from secure and both being used in the same sketch. The best way to solve this is really having Arduino API changed to include two APIs or have the library smart enough to use a custom one for Esp. Which then means we should expose the SecureRandom in some way still for when you need to mix.

[holgerlembke]

If someone is into security, he will for sure research how random() works on this platform. Every time after an update/change. So document the behavior and done.

If I call randomseed(), there are two situations: with a fixed value or a "pseudo random" value. both init the sw random generator. From the fixed value I expect repeatable numbers every time when calling random(). from the pseudo random seed value I expect it to be random. Both situations are granted when the first call of randomseed() turns off the hardware random generator and switch to the old software rg for following random()-calls.

If I only call random() and never randomseed() I obviously didn't care.

Expose the underlaying functions, too.

[Links2004]

I think we should make it as secure as possible, most users not think about how secure / real random is the random number they get.

The problem starts when the not really random behavior is used as "function", for example to generate every time the same "predictional" numbers after reboot. but unfortunately the API docs from arduino call it a "feature" that the random is not random. https://www.arduino.cc/en/Reference/Random

i can think of two concepts:

to maintain best compatibility to Arduino: default: HW rnd if randomSeed is called with the same value as last analogRead(A0) keep HW rnd. if randomSeed with something other or analogRead is never used use SW rnd. this has a low change to use HW in the wrong cases and still uses HW rnd.

most security (prefered one): default: HW rnd only switch to SW rnd when the user call the set API. but this may give use some Issues later one for some specific use cases (I think there are not so many out there). I personal prefer this be course i like it to be secure as possible by default.

Independent of this we can add a functions for HW and SW rnd. then all libs that are made for the ESP8266 can use what they like without affecting libs that use the standard interface. then the biggest problem is what happen when there are too libs one that need real random and one that needs "predictional" random values. and a function for the user to switch if he likes.

[Makuna]

@Links2004 You solution on your first suggestion won't work. Not everyone uses analogRead(A0) directly and if you did it never worked that well. (see GitHub\Makuna\RandomSeed as example of a better seeding and a method that breaks your assumption here).

[Links2004]

yes more advanced seeding processes are not detectable, but its the best of what i can think too keep it compatible to the Arduino API. If you use a more advanced seeding the "compatible solution" will fall back to predictional random mode. I not prefer this but the only way I currently can image where we have a change to use the HW as default, and can be 99,902% compatible to Arduino when randomSeed is called. if randomSeed is not called its no problem like @holgerlembke noted before. if you have some other Ideas for it we are open for it ;)

[Makuna]

We should try not to break Arduino APIs or break default Arduino assumptions.

If calling randomSeed() with the same seed value EVER returns a different series of random numbers "by default" we have BROKEN the Arduino API. Unless its unavoidable due to situations outside our control, I believe we can't do this. It is documented as a feature and almost every OS out there, random is documented and supported this same way.

The random api was never meant to be secure; most OS API expose a second API for encryption level random and point developers to it.

[skorokithakis]

If compatibility is a goal (sounds like it is), then I agree with having a configuration option, like setPRNG(PRNG_COMPATIBLE), setPRNG(PRNG_SECURE). Those names are better indicators of what each does, and people will easily find more information after some research. I believe defaults are important, so I would prefer it if the PRNG defaulted to PRNG_SECURE (HW RNG) with randomSeed setting it back to PRNG_COMPATIBLE so values could be predictable. I think that would be the best of both worlds.

Is there no way to implement a better SW PRNG, like Mersenne Twister or SFMT? Initializing that with the HW RNG would be an even better "compatible" default.

[tablatronix]

What holgerlembke and makuna said. Secure random, reproducible and bakcward compatible seed behavior.

If i call random i expect random, if i first called seed then i expect special random, if i seeded with some random source then i probably didnt read the docs or expected something better that was never actually true.

[igrr]

2142 implements proposed changes.