A general API helper programming language

mjarkk commented 4 years ago

Problem

To better explain the project description i firstly need to tell the problem i have.
So i wish to write a native app for every platform but this takes a lot of time to do because every platform works just a bit different and thus requiring a lot of code for a lot of the same things.
The biggest issue to me here seems to be the code between the API call and the application deciding that something should be send to the API like pressing the save button.

To solve this i started using GraphQL and it helps somewhat by making it easy to know that the API is capable off but i noticed that there is still a lot of things required to get from pressing a save button to updating something or getting just data on the screen that is logically the same on every framework/language.

So here are a few of these things that are logically always the same.

The GraphQL query
The response type data
Transforming filled in data to request data

These are just a few things but when you have to maintain 2 or more apps/programs with thousands of API calls this can start to take lots of time if adding or updating things.

Project description (read Problem first)

I wish to have a programming language in between of the application and the GraphQL schema wherein i can write the actions that can be dune using the API that can be compiled into Typescript, Go, Rust, Swift, Java, ..etc code.
The compiled code must work like a library so it's easy and the border between the generated and native code is clear.

I would imagine the code would look something like this:
(The end user can compile this code to a language of their choice )

// The user must import the GraphQL scheme so we can use multiple schemas
import schema from schema.graphql;

// Define add functions / variables that are language native and must be defined by the library user if they the want to use the compiled code.
define fetch (query: query<output: QueryResults>) -> QueryResults || error;
define createSlug (name: string) -> string;

// this is a function that creates a new post
// pub tells it to be public and available to the end user
// fn tells it's a function
// and MakePost is the name of the function
pub fn MakePost(name: string, description: string) {
    var slug = createSlug(name);
    var newPost = schema.methods.MakePost(name: name, slug: slug, description: description) {
        name
        description
        updateDate
        createDate
    };
    return fetch(newPost);
}

If compiled to typescript this should work something like this:

import API from 'api_code/api.ts'
import create_slug from 'x_slug_library'

const api = API.new({
  // The user must provide fetch and createSlug because those are requested by the program
  fetch: (query: string) => {},
  createSlug: (name: string) => create_slug(name)
})

const createdPost = api.MakePost() // The createdPost is an interface type with name, description, updateDate and createDate
console.log(createdPost.createDate)

What the language should be and must be able to do:

Typed
Very minimal language so it's easy to compile it to x language
no framework / language specific things having it use x library by default in x language makes it usually incompatible with a lot of project and thus users should write their own fetch, or x function
GraphQL inputs and types must be transformed into native types if the language has types.

Relevant Technology

GraphQL

Complexity

[ ] Beginner - This project requires no or little prior knowledge of the technolog(y|ies) specified to contribute to the project
[ ] Intermediate - The user should have some prior knowledge of the technolog(y|ies) to the point where they know how to use it, but not necessarily all the nooks and crannies of the technology
[x] Advanced - The project requires the user to have a good understanding of all components of the project to contribute

Required time (ETA)

[ ] Little work - A couple of days
[ ] Medium work - A week or two
[x] Much work - The project will take more than a couple of weeks and serious planning is required

TheOtterlord commented 4 years ago

Hello @mjarkk.

I think that you should have a look at Flutter. It is Google's mobile sdk programmed in Dart. They are currently working on web & desktop support too. If you are looking for something other than app creation, Dart compiles into many languages and has lots of plugins on pub.dev.

mjarkk commented 4 years ago

@TheOtterlord Thanks for the suggestion but that's not necessary what i want.
It indeed allows me to make native looking apps that are resalable and "fast" but it's not native code.

My problem with flutter is that it's kinda limited to 1 style app, if you want to have the best apps you still want to write native apps that are as native as they can so on windows i want a uwp app because that's what windows wants you to do. On linux i want an GTK app because that gives the best linux looks and so on..

With 1 style i mean the app just has a certain look and feel, yes flutter has styles but that doesn't mean the overall style of the app changes.
Take for example whatsapp it has a completely different look on android vs ios. If it was written in flutter i would think it would look a lot more equal on android and IOS.
I personally can live with that but now also add a desktop app into the conversation the flutter desktop apps don't look native at all they look a lot like electron apps, that's fine if that's someones style but not everyone wants a quick app some people want a real native app like me.

Now it's true you can compile dart into some other languages but there are not a lot of them (js and a binary executable so far i can see) beside that dart is not meant to be converted into lots of other languages.
Even if it could it would result in bloated complicated code and that's not great.

For a good example take Plex they have native apps for almost every TV os, consoles, desktop, phone, web and they all have a native look and feel to them.
Something like that can't be dune with flutter due to the reasons above.
That's why i would like to use native code but that takes a lot of time to maintain so it would be nice if there is some kind of extra abstraction that handles things all of the clients do like form checking, sending it to a web server, etc.
To do this i see 2 solutions either write all the code that all clients do in a language that can be imported by most languages or have some language that compiles into the platform it's native code.
The first solution should work because almost every language has support for importing webassembly or c/c++ code, buttt... for platforms like the web you don't want to ship a 5MB blob if the API changes often.
So this leaves me with the last solution of having some kind of language that can be converted into native code so i only have to write the business logic once and after than i'm able to use it in all languages natively.

TheOtterlord commented 4 years ago

Fair enough. Flutter does not nearly have as many designs as I would like either. It seems that making a new language could be needed as compiling C++ for the web would not work. Maybe the app class/object could contain a theme parameter. The default would be "native", but people could extend a theme class to make their own or set it to one of the already built ones that we need for platform compatibility.

Because we would not only have to create a programming language, but also libraries for each supported system, we will need a much bigger team. I do not mind working on this, but I do have limited time. I can program is JavaScript, Dart, Python, and know minimum C++ and Java. I have created a programming language in Python before, but never finished it. It was very similar to JavaScript.

My last sentence makes another point. How native would our language be. We are supporting low level languages such as C++, but also languages like JavaScript. This raises issues with some parts of the native language compilation vs the web compilation.

I just want to point out quickly that Flutter is open-source and has also got cupertino style (iOS) support. So we could always piggyback off of that and just write native looking themes by either editing the Material library or creating it from scratch.

As I said before, I am happy to work on the project, but we will need more support (unless we make flutter libraries).

mjarkk commented 4 years ago

Because we would not only have to create a programming language, but also libraries for each supported system

I image it to work similar to openapi-generator where you have 1 thing in their case being the open api schema and in our case the api code and you transpile it into x programming language files.
There are i think a few parts to this:

Compiler
1. Tokenize the code
2. A library for every language that uses tokens to convert into other code
Editor support

we will need a much bigger team

Not sure, i think it's not that hard if the programming language made is simple especially as in being convertible to other languages.
To keep it simple there should be a few things that we don't want.

It should never be able to do any system specific actions like fetching because there are a lot of ways the same thing can be achieved and such things are up to the end user.
We should heavily relay on types as a lot of languages also use them.
Also at the start things like generics and classes should not be implemented.

How native would our language be. We are supporting low level languages such as C++, but also languages like JavaScript.

Yes, My thought was to support as much languages as we can.

This raises issues with some parts of the native language compilation vs the web compilation.

Yes this is indeed an issue and that's why the language must be simple and never use system functions like fetch.
That said i think it should be designed for low level languages because then it will be not to much work to also support high level languages. The other way around will cause meany problems i think.

I just want to point out quickly that Flutter is open-source and has also got cupertino style (iOS) support. So we could always piggyback off of that and just write native looking themes by either editing the Material library or creating it from scratch.

True and this will defensively happen once flutter also becomes popular on the desktop especial with windows and their UWP. I'm totally in for adding styles like that to flutter (Can't wait to see some sexy UWP style windows apps written in flutter) but there are still issues like those once i explained in my last comment and a sexy windows style UWP app will look like garbage on linux & macos.

As I said before, I am happy to work on the project, but we will need more support

Ya i might be feeling it's simpler than you pointed out but it's not a weekend project nor can i complete it in a month beside my day to day work.

If i would start this project myself i would think about writing it in either Rust or GO because i both like them, have enough personal experience with them to feel confident enough to write a compiler/transpiler, they are both fast go might be garbage collected but when making proper use of the go routines it can be supper fast, they are both modern languages still on the rise in popularity and at least they both compile into binaries and all compilers i know are binaries so i think it's best to do the same.

TheOtterlord commented 4 years ago

If we make the language simple enough, we may end up missing out on some native things that some apps need. We could fix this by adding the ability to add native code (especially during development) using a specific function in our API. Though this would probably be decided later.

I imagine we would be supporting types with things like int or str because of the lower level languages requirements. I have not worked in lower level languages that much, so I may be wrong.

Rust looks quite good, not heard of go but I'll have a look. They both seem to compile into the big 3 which we will need for constant support. I think that after we finish a prototype, we should test it by building an IDE. This will help to show it off to interested developers, and we probably want and IDE anyway.

With Flutter, the desktop build commands will be different depending on the system, meaning you can build seperately for macOS, Windows and Linux. We could build seperate libraries for each design system (macOS, UWP, GTK3, etc...) and people could build apps accordingly.

Finally, in terms of language structure and syntax, I really like the tree layout of Flutter and had a premature design of an app file.

` import our_api;

App app = App({ title: "Hello world", body: Body( children: [ Text("Hello world!") ] ) });

app.buildWin(); app.buildLinux(); app.buildMac(); app.buildWeb();

// testing // app.debugWindows(); `

I don't have a preference when it comes to Flutter libraries vs custom language + API, so since it is your idea, let me know which one you prefer and we can get started.

TheOtterlord commented 4 years ago

I have had a go at both languages, and I believe golang would work better. It is more flexible, which is important when creating our tokeniser.

TheOtterlord commented 4 years ago

I've created a simple tokenizer in Golang, just to test my skills.

// change to tokenize when creating compiler
package main

// imports
import "fmt"
import "strings"
import "regexp"

// The token structure needs only the name of the token and the word
type Token struct {
  name string
  word string
}

// tokenizes the source code
func tokenize(src string)  {
  // split src into words
  src = strings.ReplaceAll(src, "\n", " \n ")
  src = strings.ReplaceAll(src, ";", " ; ")
  words := strings.Split(src, " ")
  // define tokens
  tokens := make([]Token, len(words))
  count := 0
  // for each valid word: create a token
  for i := 0; i < len(words); i++ {
    word := words[i]
    if word != "" {
      // define regex
      var isIdentifier, idErr = regexp.MatchString("^[a-zA-Z0-9_]*$", word)
      var isInt, intErr = regexp.MatchString("^[0-9-]*$", word)
      // find type of token
      if word == "\n" || word == ";" {
        tokens[count] = Token{name: "NEWLINE", word: word}
      } else if word == "//" {
        tokens[count] = Token{name: "LINECOMMENT", word: word}
      } else if word == "/*" {
        tokens[count] = Token{name: "OPENCOMMENT", word: word}
      } else if word == "*/" {
        tokens[count] = Token{name: "CLOSECOMMENT", word: word}
      } else if word == "func" {
        tokens[count] = Token{name: "FUNCTION", word: word}
      } else if word == "const" {
        tokens[count] = Token{name: "CONSTANT", word: word}
      } else if word == "var" {
        tokens[count] = Token{name: "VARIABLE", word: word}
      } else if word == "=" {
        tokens[count] = Token{name: "EQUALS", word: word}
      } else if isInt {
        tokens[count] = Token{name: "INTEGER", word: word}
      } else if isIdentifier {
        tokens[count] = Token{name: "IDENTIFIER", word: word}
      } else {
        tokens[count] = Token{name: "UNKNOWN", word: word}
        if idErr!=nil || intErr!=nil {
          fmt.Println("A regex error occured");
        }
      }
      count++
    } // else ignore word
  }
  // reduce tokens to valid tokens
  final_tokens := make([]Token, count)
  for i := 0; i < count; i++ {
    final_tokens[i] = tokens[i]
  }
  fmt.Println(final_tokens)
}

func main() {
  // test with some mockup code
  tokenize(`// This is a comment. It ends with a line break. Ignore when parsing
    /*
      Multi-line comment
      Remember to ignore everything here when parsing
    */
    var test = 0;
    var test2 = -0;`)
}

It works pretty well for a prototype, though I have not written any tests yet. And this is just a concept of the language.

It has occured to me that we are getting to the point where we need to begin planning, coding and documenting.

Assuming we would not be building Flutter libraries, this would be a multi-repository project. We will need a repo for the compiler, and a repo for the API. We are going to need names for each of these, and while I have no idea what to call the language, I did have an idea for the API. It's not great, but just an idea to get us thinking.

UNPAIN UNiversal Programming Api Ingeniously Native

Told you, not great. Anyway, once we have a name, we can create a repo.

mjarkk commented 4 years ago

@TheOtterlord Thanks for all the comments!

If we make the language simple enough, we may end up missing out on some native things that some apps need. We could fix this by adding the ability to add native code (especially during development) using a specific function in our API. Though this would probably be decided later.

Yes, my idea to do that was via code like this in the programming language as also described in the original issue body:

define fetch (query: query<output: QueryResults>) -> QueryResults || error;

If lines like this are added and the user wants to use the transpiled code they must first define these functions in their language (dunno how our transpiled code should read these but this is also not a too hard of a problem to figure out).

I imagine we would be supporting types with things like int or str because of the lower level languages requirements. I have not worked in lower level languages that much, so I may be wrong.

Yes defensively.

we should test it by building an IDE. This will help to show it off to interested developers, and we probably want and IDE anyway.

Not sure if we should build an IDE but neither do i have good ideas to showoff it's potential so why not :).
I'm not sure if we want to build our own IDE just for writing this code i rather prefer good support in already existing editors like VS-code because most people already use those.

I have had a go at both languages, and I believe golang would work better. It is more flexible, which is important when creating our tokeniser.

I'm not sure if i agree here Rust might look and feel like at first but it's language features are way more than go. Personally i have the most experience with GoLang as my daily work consists of writing lots of it :) but this also means i know the limitation of go and there are meany compared to languages like C++ or Rust.
But golang does one thing very good and that is making concurrency very accessible compared to almost all languages and this is kinda important for a compiler though in rust someone also do concurrent things but it's more classic concurrency as dune in languages like C++ or Swift.

It works pretty well for a prototype, though I have not written any tests yet. And this is just a concept of the language.

Ah testing is also a very important point that i haven't thought about it yet.
I must admit i usually don't write tests for my project :see_no_evil: but we should definitely for this. Go and Rust have builtin tests libraries and i like the rust tests the most as they are supper easy to create and overall just a bit more wellcomming that go's tests.

It has occured to me that we are getting to the point where we need to begin planning, coding and documenting.

Ai it seems like :). The upcoming weekend i will start to write some code for this and see how far i can come.
I don't want to make any plans yet for creating orgs naming the project etc.. for now i just want to see if it's even possible by creating a simple transpiler.
I also think we have to little data now to even create some documentation.

TheOtterlord commented 4 years ago

Cool, I'll keep working on my tokenizer for practice. I'll also try to parse it into JavaScript.

Let me know how you get on ;-)

mjarkk commented 4 years ago

HeyHey i've worked this weekend a bit on a tokennizer and made some nice progress but still lots and lots to do of course.
i think i will make my code public once i'm at the point where i can start writing the part that transform the tokens into other code like typescript, go, rust, etc..

~~Til that point i will keep a TODO list in this comment up to date.~~ See the temp repo for a todo list: https://github.com/mjarkk/general_programming_language#roadmap

TheOtterlord commented 4 years ago

Ok. Looks like your making good progress. I'm happy to parse into TypeScript once you've set up the repo. Let me know if there is anything you need me to do in the meantime ;-)

mjarkk commented 4 years ago

That would be amazing!!
I'll keep you up to date.

TheOtterlord commented 4 years ago

Quick question.

What language are you using for the tokenizer? I feel like writing some tests based on your schema above.

mjarkk commented 4 years ago

Rust.

Currently i write tests for the code i'm working on but you could add more if you want to.
Somewhere this weekend i'll put my code in a repo then you can easily write some more tests if you want to :)
The tests are currently supper easy and only exists out of a string that needs to be parsed successfully not checking if the parsed content is actually correct.

mjarkk commented 4 years ago

I've created a temporarly repo: https://github.com/mjarkk/general_programming_language
If we are at the point where we can compile a simple file to some other language i think we can create a github organization for this project
Feel free to fork it and modify the code.

TheOtterlord commented 4 years ago

Cool. I'll have a look and try to create some validation tests.

mjarkk commented 4 years ago

Hey @TheOtterlord ,

I'm during the week quite buzzy with work so you won't see me working to much on this project during work days.
Maybe it's a good idea if we have a slack, discord, etc... where we can communicate simple things more quickly for this project :)
Do you have any suggestions for message any messaging platforms?

TheOtterlord commented 4 years ago

This is a bit early, but once we create an organization, we have access to team discussions which will allow us to communicate and also seperate discussions about different things. For example, we could have one about work on the programming language and one about the API. I think this would be the best option, once we create the team. For now though, something else is needed.

I'm not on slack or discord, but I did glance at gitter which seems to work quite well.

It is not as fast, but GitHub issues also give the ability to seperate discussions in the open. I have seen other OSS use this to have weekly meetups that are archived in the open for users to see.

mjarkk commented 4 years ago

Ah i didn't know about the team discussions, it seems like a good option as it is on the same site as the code.
I suppose we do that once we have a github organization.

TheOtterlord commented 4 years ago

Yeah, it should help to keep everything in one place. Should we use something else in the meantime, or just use this issue?

mjarkk commented 4 years ago

sure lets use github issues to communicate in the mean while.

open-source-ideas / ideas