Are we @nogc yet?

[atilaneves]

Description

It's common to see potential D users comment that large parts of the standard library aren't usable with the GC. It's difficult to counter that assertion without any data - I don't think anyone knows how much of Phobos can be used from @nogc code, nor could we point people to a resource for more information.

Similarly to Python's former wall of shame that got renamed to the wall of superpowers, it would be great if D had a webpage titled "Are we @nogc yet", preferably with a code-coverage-style visualisation of Phobos. At the very least, a list of functions/structs/classes that are @nogc compatible.

What are rough milestones of this project?

• Annotation of Phobos unittests with @nogc if possible
• A tool to extract the information of what Phobos code is called/covered by such unittests
• A web page to visualise the data

How does this project help the D community?

It helps the community by helping to dispel the myth that the GC is central to D programming.

Recommended skills

Webpage design and implementation.

Rating

Easy

Project type

Low hanging fruit, Fun

Point of Contact/Possible mentors

@atilaneves @maxhaton

[wilzbach]

Such a tool could also similarly do this for the @betterC attribute and @safe attributes.

[atilaneves]

@betterC is a good idea. I think @safe would be too, except that we plan to transition to @safe by default.

[PetarKirov]

@atilaneves I agree that this is a important project for both the evolution of D.

Showing "@nogc ready" next to each template function with a @nogc unit test, in addition to a website tracking our progress can go a long way to improve the perception of the language, in addition to helping drive contributors on this pain point for some (potential) users.

However, I'd like to clarify our strategy as to making functions @nogc in practice.

There several varieties of functions (please correct me to make the list exhaustive):

1. Verified @nogc by unit tests
2. De facto @nogc, but not verified yet
3. Can be made @nogc with little changes.
4. Not @nogc, because they use enforce or similar
   • Some uses of enforce are actually checking for programming errors and can be replaced with assert
   • For the rest DIP1008 can help
5. Need to allocate temporary memory, which can be disposed by the end of the function. We can use:
   • static arrays when the memory requirements are of bounded and reasonable size
   • std.internal.scopebuffer or similar for the other cases
6. Need to allocate dynamic, non-temporary, but internal memory, for example: std.functional.memoize, std.algorithm.iteration.permutations, etc.
   • We can use std.container.array, which is already @nogc
7. Functions (including constructors) that return containers or objects composed of containers that need dynamic memory

So, what do we do about 6. and 7.?

---

For some time the plan was:

1. Add allocators (completed in Nov 2015)
2. Add smart pointers and other containers (we have various libraries avaiable on code.dlang.org, but nothing "standardized" in druntime or phobos)

3. (option A) Add overloads to functions that need to allocate memory, for exmaple

   // before:
   auto transmogrify(Args...)(Args args)
   
   // after:
   auto transmogrify(Allocator, Args...)(ref Allocator allocator, Args args)

   (option B) Don't change the API, but use std.experimental.allocator.theAllocator internally and handle memory by RAII objects, like containers and smart pointers.

   (option C) Use IAllocator, ISharedAllocator or RCIAllocator in case a function can't be templated (e.g. virtual functions, extern (*) APIs),

But we didn't follow up on that, since leaking std.experimental.*, into the rest of std.* (or even core.*) was kind of a taboo. It didn't help that the choice between option A, option B and option C (probably I'm missing some other option) is not very clear cut.

---

Here's my proposal:

• move std.experimental.allocator to core.memory.allocator and move GC to core.memory.gc (and have core.memory package for backwards compatibility)
• Make std.container.array allocator aware
• Split the GC implementation into allocator building blocks (so they can be available under a uniform API)
• Add UniqueRef and LocalSharedRef and ThreadSharedRef smart pointer types to core.lifetime
• Add struct based sync primitives (basically move the implementation bits of core.sync.* into structs and reuse them inside the existing core.sync.* classes)
• Refactor core.thread to expose @nogc low-level threading primitives and use them underneath the existing Thread and Fiber classes
• Start adding allocator aware overloads to phobos

[wilzbach]

@PetarKirov I like most of your points, though I do have a few comments.

Add smart pointers and other containers (we have various libraries avaiable on code.dlang.org, but nothing "standardized" in druntime or phobos)

It was planned to do this and there was quite some work done. It failed, because no one could agree on the implementation of RC. A bit of history:

https://github.com/dlang/druntime/pull/2760 https://github.com/dlang/druntime/pull/2690 https://github.com/dlang/druntime/pull/2679 https://github.com/dlang/druntime/pull/2646

move std.experimental.allocator to core.memory.allocator

I am not sure whether this is a good idea short-term wise. There are a few problems with std.experimental.allocator's interface, the biggest is at least the Phobos version can't be used in @betterC (the stdx-allocator fork on dub can since v3). more issues. For the collection project it became clear that you don't want a PureNotSafeAllocatedString type, but when you don't customize the type you have two options: a) make the IAllocator interface very restrictive (e.g. pure @nogc @safe nothrow). Problem: almost no allocator provides there guarantees b) make the IAllocator interface open (e.g. @nogc). Problem: no type safety benefits and you loose the benefits of having e.g. an @safe allocator.

As there's also a cost in carrying the pointer to the allocator around, it was thus decided that at least all the upcoming core.rc data structures will use sth. similar to theAllocator which implements (a).

Make std.container.array allocator aware

Again, I am not sure this is a good idea. 1) I don't think this would work without many breaking changes. One example, you can slice Array and return a range. This allows the user to pass this copy to anywhere - no scope limitations or reference-counting here. 2) You want to have @nogc data structures in druntime for many reasons (rcstrings, betterC support, ...) 3) RC collections are even ready (e.g. https://github.com/dlang/druntime/pull/2760). They just need a consensus on core.rc.

Start adding allocator aware overloads to phobos

This would be a massive project and I'm not sure people wanna set an allocator whenever they do e.g. dirEntries. I think arrays (and thus strings) + associative arrays being GC-allocated is probably 80% of the @nogc problems in Phobos. I don't think that people want to manually free every string/array etc. that they allocated, so you probably want to use an RCAllocator for most cases. However, you can't change the return type of functions, because this would break existing code. tl;dr:

• templated functions with allocators -> massive overloads + usability problem (as the allocator needs to be passed to every function)
• non-templated -> manual free
• RCAllocated -> break of return type

IMHO we need to have strings and array which are @nogc, before we even think about improving Phobos. However, even with an rcstring and rcarray type right now, it would still be hard to change existing functions in Phobos without breakage, so that we're essentially looking at Phobos 2 either way. The only way I see to do this in a non-breaking way if the user could define the allocation of arrays for the entire program (e.g. via compiler flag --alloc=rc).