Project system

[flaxed]

The discussion on File References #9 is starting to overflow into how the project system will work.

The term Project System in this case is meant to be broad.

There are several options to implement this:

• Visual Studio like projects
• Root folder marked by a dot file
• Arbitrary with a list of files passed directly to the tools

Since one of the aims of this project is to avoid conventions, this limits the pratical ways to implement a solution.

From previous discussions the solution with arbitrary list of files would be the one with least impact, allowing for additional tooling to target the project system to any existing conventions.

[leiradel]

4. Arbitrary list of folders where the tooling will look for .dll files when it finds import statements, just like the -I flag that is common in many C/C++ compilers.

[flaxed]

That is similar to option 3. Don't see an issue with extending it to support folder paths.

Like you've said before, both 3. and 4. limit how the parsing works, forcing to parse every file for defined types. But maybe that isn't an issue for now. An initial PoC could just brute force it, and ahead we can develop cache mechanisms, or even a package system with an explicit list of provided types and namespaces.

[leiradel]

Option #4 only tells the tooling where to look for files, it doesn't mean you have to grab and parse all files starting on the given folders.

I.e. if you pass -I /home/user/proj/ddl and compile a file with only import math.vectors;, the compiler will load and parse /home/user/proj/ddl/math/vectors.ddl and won't touch any other files.

[flaxed]

I understand that, but when you say that import math.vectors makes the tooling look up in /math/vectors.ddl that is a convention.

Probably is the most straight forward to understand, but then it also limits namespaces to be determined by file structure. Personally don't think it's a bad approach, also used this in rust and is easy to grasp. But if this project aims to be inclusive, its the sort of solution that we shouldn't jump straight into, so we don't leave potential projects behind with this approach.

One option would let the tooling be configurable. Allow for the project to choose if they want brute force, path based convention, etc. Wouldn't leave projects behind, but impacts namespacing, making it inconsistent across projects.

[leiradel]

It works just like the C preprocessor, so it's a win for me. This should be simple to understand and use.

[leiradel]

The most generic solution would be to have some kind of callback that is called whenever an import is found and the compiler needs to resolve a file path. People could use this to program any behavior they want.

If we provide a default callback that implements a convention it will be a success and no one will bother writing their own. If we don't provide a default callback it will be a failure because of the high cost to get started.

[flaxed]

I like this callback solution a lot! And you touch on the points that I see essential for a succesful project.

One issue that might arise from this is the divergence across different projects with different callback implementations. Say project A uses the default implementation, and project B uses a different implementation. You won't be able to simply package the .ddls in project A and send them to the B team, without first refactoring them. For this use case I'd suggest that we investigate a possible package format. Not talking of a whole package distribution system like npm, just a container format that will package the relevant .ddls and a type list, and that can be referenced in a standard way. This referencing wouldn't support callback customization.

Final point is the expression callback. It will sound nit-picking, and probably is :), but I think naming is important. I'd prefer we use something like plug-in system, tho I feel this is too heavy of a term for the functionality. Maybe just call it reference system? default reference system and custom reference system don't sound too bad.

[leiradel]

The callback would be a deal breaker for me. This and the project system. I want something that just works and have zero friction when I want to share stuff with other people.

Every developer knows -I, DDL's tooling should just use the same flag to resolve imported files.

[flaxed]

I don't understand, how is the callback a dealbreaker when you've said that a default callback would be a success?

As I see it there are three things to consider here:

1. Default Behaviour

First, as you've said, we need a good default behaviour. For default behavior I'd suggest https://github.com/TheToolsmiths/ddl/issues/9#issuecomment-524078829 which is similar to your -I suggestion.

You have the math file /libs/foundation/math.ddl with

def struct Vec3(float x, float y, float z);

and a source file with src/units/unit.ddl with

include math;

def struct Unit
{
 position: math::vec3,
}

And then the compiler would be called with

dll compile "src/**/*.ddl" -I "/libs/foundation"

This would be the default behaviour and if you run the compiler as shipped, it would use this.

2. Custom Behaviour

Second is how we can accomodate projects that already have a set structure that can't work with this way to resolve references. For this I like your callback suggestion. Using the unit.ddl example above, the compiler provides the reference system a string with "math" which returns a list of files, including /libs/foundation/math.ddl. With this list of files, the type system can parse them and pick the right type.

As I type this I've noticed an issue we'll have, with both default and custom callbacks, that is how to desambiguate from types with same name from the same import statement?

3. Consistent cross-project type sharing

Assuming both points 1. and 2. bring value to the project on their own, which I believe they will, it's also fair to assume that 1. and 2. aren't directly compatible. And not being compatible means there will be a divergence of projects using 1. and their own version of 2., and even two different projects with different 2. solutions.

Thats why I feel it is important to separate from files inside the project and files meant to be shared. The latter should be as consistent as possible, so any project with solution 1. or 2. can export types and share them to any other project, without worries of refactoring and resolution issues.

[leiradel]

I don't understand, how is the callback a dealbreaker when you've said that a default callback would be a success?

There's no default callback, since there's no callback. There's only the -I flag, which defines a root path from which the tooling will search for imported files.

Second is how we can accomodate projects that already have a set structure that can't work with this way to resolve references

They don't use this DDL, since it doesn't exist yet. When it does, it's just a matter of structuring your project according to the tooling, just like what C/C++, Java, and other projects are structured.

Consistent cross-project type sharing

Lua 5.2 dropped locale-dependent identifiers because it made it difficult to share scripts, since identifiers valid in one locale could be invalid in another, different locale, which leads to confusion and frustration.

Imagine if C compilers adopted callbacks to resolve included files. Imagine someone integrating source code from multiple libraries, each one using its own way to do that.

[flaxed]

I don't understand, how is the callback a dealbreaker when you've said that a default callback would be a success?

There's no default callback, since there's no callback. There's only the -I flag, which defines a root path from which the tooling will search for imported files.

If the next point is valid, there is a need for a callback, and therefore there will be a callback.

Second is how we can accomodate projects that already have a set structure that can't work with this way to resolve references

They don't use this DDL, since it doesn't exist yet. When it does, it's just a matter of structuring your project according to the tooling, just like what C/C++, Java, and other projects are structured.

Doesn't matter if the DDL exists or not, this blocks any project from adopting the DDL without work to conform their files to this convention. Don't think that it is acceptable to leave projects behind just because they don't or can't conform to the convention, especially when the project is starting and we can easily add a way to ease adoption.

The reason I see this deserving of focus is because it seems one of those features that you can't just tackle on after a few people complain, if they complain at all and just don't ouright dismiss the format. Easier to remove this on a later version if it ends up adding issues.

C++, Java and other languages have their own ecosystems and are considered the main development framework. This DDL and its tools will be a supporting tool, and as such need to be able to adapt to the needs of the main project.

Amongst other things, this is what being inclusive means.

Consistent cross-project type sharing

Lua 5.2 dropped locale-dependent identifiers because it made it difficult to share scripts, since identifiers valid in one locale could be invalid in another, different locale, which leads to confusion and frustration.

Imagine if C compilers adopted callbacks to resolve included files. Imagine someone integrating source code from multiple libraries, each one using its own way to do that.

One of the few things at the time in the wiki's specifications is that the format will use UTF-8. So locales by themselves shouldn't be an issue.

If your example is so that we don't fall on similar mistakes of having context dependent parsing, I understand and is a valid point to worry about, but still don't think it's worth price of forcing hard conventions. Besides, if we give everyone the tools to only require the callback solution inside their own projects, it will an issue limited to their own projects. Shoudn't leak out.

I feel like that hopefully we're aiming to the same general direction, so let me explain my whole thinking:

• I agree that -I will be a good default approach. We should push everyone to use this solution.

• If they for some reason can't, then they can use the callback extension system, BUT I only see this acceptable for intra-project references, and we should also make that clear. Also, this will be OPT-IN, so you'd only use it if it bring value to your project.

• Since we're allowing the use of a callback system, we should also make it easy and simple for users to share their files across projects. This tooling would take the files that need to be exported and transpiles them to have the default convention. This way the custom callback convention won't leak out of the project, and whoever references these exported types will have to use a default convention.

I hope that this explains my standing a bit better.

[leiradel]

Also, this will be OPT-IN

And then you want to share your files and regret having a non-standard setup.

This tooling would take the files that need to be exported and transpiles them to have the default convention.

Please no. Simpler is better.

[flaxed]

Also, this will be OPT-IN

And then you want to share your files and regret having a non-standard setup.

Again, that’s the whole idea of exporting to sharing.

This tooling would take the files that need to be exported and transpiles them to have the default convention.

Please no. Simpler is better.

Simple is better if it fulfills it’s purpose. If it doesn’t, then it isn’t simple, it’s lacking.