Expose the null stream in cust

[kjetilkjeka]

Are there any reasons for currently not exposing the null stream in cust?

Would there be any problems to implementing Default::default() for Stream as the null stream?

A lot of traditional C++ cuda code uses the null stream. I think exposing it in rust as well can lower the complexity of having to manage streams yourself. It can also make the on-boarding experience close to what it is in C++.

[RDambrosio016]

This is mostly based off opinion but also facts, i chose to not expose null streams (well, to keep what rustacuda was doing in not exposing them) due to several reasons:

• null streams are massive footguns, everything being sync is terrible for performance, users should know what streams are instead of being blissfully unaware of them and likely getting worse performance that they could be getting.
• null streams implicitly synchronize with real streams unless the stream is NON_BLOCKING, however, NON_BLOCKING is unsound because of weird memcpy behavior where a HtoD memcpy does not happen sync, it actually happens async in the null stream, fun isnt it?
• Using the null stream complicates things for anyone using your crate upstream, its impossible to run your crate kernels async and/or overlapping if you do not expose a way to use streams.
• It complicates the API, null streams are treated in a special way by many things, a drop-in Default null stream would likely not work at all.
• Creating a stream is so easy that you may as well just make one.
• as mentioned, streams implicitly synchronize with the null stream, which makes it infectious, using the null stream significantly hinders the ability of a user to control exactly how commands are enqueued on the GPU.

I believe null streams were a big mistake in CUDA, they were made just to simplify launching kernels, but they hide so many details under the surface that at this point they should be phased out.

[kjetilkjeka]

I understand this is a difficult decision. If the motivation is to create an opinionated library for using the GPU to accelerate Rust code the approach makes a lot of sense.

To help you understand where I'm currently coming from. We're currently looking into writing one of our applications in Rust. We will need to use computations on the GPU in our application. We have a strong C++ team that is interested in Rust, but is not willing to introduce any technical risk. This means we might use Rust for host code and Cuda C/C++ compiled to ptx on device.

Due to the early state of Rust GPU it makes sense to, for now, learn from the patterns as we use in C++. This means we need to be able to reach for all/most parts of the cuda runtime library. I'm not sure if there's any way to unify our concerns? If not I think it might be a place in the ecosystem for a non-opinionated library attempting to expose most Cuda functionality as well.

[RDambrosio016]

Streams are not a concept exclusive to rust-gpu, any good cuda application uses streams because the null stream is bad for performance when you want to run kernels async or multiple kernels. You lose nothing by using explicit streams over null streams. You don't even lose interop capabilities because streams sync with the default stream implicitly. So i don't really understand your concern, explicit streams only gain more things you can do, they don't lose any capabilities. This gets even more complex when you factor in legacy default streams vs per-thread default streams. It is so easy to shoot yourself in the foot unless you are deeply familiar with the details of the driver API.

[RDambrosio016]

One of the major goals of this project is clearing away the legacy cuda cruft and moving on to proper ways of doing things. This is exactly why:

• I have not added the basic relaxed atomic ops like atomicAdd in favor of writing atomics once and doing it correctly using modern ptx instructions.
• I have not exposed default streams and i will not ever expose them. If you want a single global stream use a lazy static.

It is critical to establish a standard for how to write CUDA applications idiomatically, and it is my responsibility to establish such a standard at such an early stage. Exposing the default stream right now would encourage people to create subpar code and ignore such a critical part of GPU programming. Therefore i will not expose the null stream, you can emulate them with a lazy static perfectly, but there is rarely a reason to do this.

Consider this, would you say the same if Vulkan had a single blocking universal queue instead of real queues?

[kjetilkjeka]

I understand your motivation and think it is an excellent goal. As a long time goal we are also very interested in finding out the most idiomatic way to use Rust in Cuda and move towards it. Even though we might not be able to use everything here due to some of our short deadlines we will still follow everything you do here tightly and hope to cooperate as much as possible.