Finalise the new typesystem branch.

[azreika]

The typesystem is currently well into being completely revamped. The current status of these changes can be seen in the new new-typesystem branch. I'm creating this issue as a one-stop location for all TODOs planned for that new branch at the moment. Feel free to comment any suggestions or enquiries below. Order is roughly planned timeline.

-- Stage 1 -- Implementing the new type-system

• [x] Implement the new type-lattice/type-analysis constructs

-- Stage 2 -- Changing syntax and semantics to work well with new type-system

• [x] Rewrite the parser to allow reasonable grammar extension
• [x] Change the component scope delimiter from . to ::, and update the tests to match
• [x] Update the grammar for records to R[...], and update the tests to match
• [x] Get ord functor test working - to fix semantic/records7
• [x] Get testsuite fully working

-- Stage 3 -- Changing type declaration syntax

• [ ] Deprecate .number_type, .symbol_type, and .type without a RHS
• [ ] Add a new universal type definition syntax - .type x = <primitive | union | record>
• [ ] Deprecate number type, and replace with int
• [ ] Update testsuite to match the syntactic changes
• [ ] Refactor code to remove assumption of only two primitive types

-- Stage 4 -- Implementing record printing [azreika/record-maps branch]

• Part 1: Single-Level

• [x] Add a record table interface

• [x] Get rough one-level record printing working for the interpreter

• [x] Get rough one-level record printing working for the compiler

• [x] Pass record table to all stream writers in both interpreter and compiler

• [x] Pass through a kind mask to all stream writers

• [x] Get one-level record printing working all the time (regardless of stream or execution type)

• [x] Fix record-printing when different-arity records are used in the same program

• [x] Merge in one-level printing properly into the souffle/new-typesystem branch (refactor)

• Part 2: Multi-Level

• [x] Pass around a type-table

• [x] Pass in type-ids as part of records somehow (add type field to RAM tuples at the start?)

• [x] Get multi-level record printing working correctly in the interpreter

• [x] Get multi-level record printing working correctly in the compiler

• [x] Merge in multi-level printing properly into the souffle/new-typesystem branch (refactor)

• Part 3: Cleaning up

• [x] Remove the warning when records are printed

• [ ] Pass only records that need to be passed

• [ ] Write it in a more optimisation-focused manner

• [ ] Add tests for record printing

• [ ] Merge final printing properly into the souffle/new-typesystem branch (refactor)

-- Stage 5 -- Implementing record polymorphism

• [ ] Allow record unions on the AST level (remove the error)
• [ ] Update the typesystem to consider record unions
• [ ] Implement record polymorphism (how? use type field from previous stage?)
• [ ] Add tests for record polymorphism

-- Stage 6 -- Writing up the documentation

• [ ] Documentation

-- Extensions -- Other considerations for typesystem improvement in the future, not covered by this issue

• Actually allow proper functor poylmorphism (e.g. ord, +, etc.) (how?)
• Add new primitive types (e.g. float, uint, and string)
• Record input fact files
• Should clean up WriteStreamCSV.h/WriteStream.h - make WriteStreamCSV a subclass of WriteStream?

[azreika]

The plan is also to eventually allow other primitive types - potentially int, uint, float, symbol, and string?

[b-scholz]

Yes - that would be wonderful to have int/uint/float/symbol.

Herbert would like to differentiate between symbols and strings.

A symbol cannot be altered - it is immutable whereas operations such as strcat are permitted for strings.

For me, uint/int/float/symbol is more important as a first step. In a second step, we can differentiate symbol and string.

[azreika]

For simplicity, current focus will be on two primitive types,int (to replace number) and symbol, with the intention of easy extension in the future to allow float, uint, and string. I think some extra refactoring will be necessary, because the assumption that only two types exist (number and symbol) is heavily built-into the code.

[azreika]

Want to potentially implement proper functor polymorphism in the future, e.g. to allow ord, +, to work correctly based on the argument types provided.

[azreika]

Printing out records will need the record map to be passed to the output engine (WriteStreamCSV, etc.). This will probably require RecordMap to be unified across the synthesiser and interpreters, similar to the SymbolTable.

Also, record polymorphism means that records in the same tree of unions require unique identifiers, which is currently not handled (at present we only guarantee unique record identifiers for records of a fixed arity). Current thought process is to give all records within a union tree a 'virtual' arity which equals the maximum real arity of any record in the union tree, and store them in the record map with that padded arity. Catch is we would need to store the real arity as an extra field in each stored record tuple.

However, we also need to store the type id with each record tuple to allow record printing later on. Current idea is to add type-id and arity as one extra field in each record tuple - 6 bits for arity (since we only allow a maximum arity of 64 fields per record type at the moment), and the remaining 26 bits for the type-id.

[azreika]

Record map unification between the synthesiser and interpreters does not seem feasible without compromising the fine-tuned performance of the record maps in the compiled version, even when records are not printed.

New idea is to have an intermediate record table being passed through to the stream-writers if records are chosen to be printed in the datalog file, where all relevant tuples in the table are copied. This means there is overhead for record map copying, but the vision is that this will only be triggered when records are printed (and only potentially-printed records are copied over).

So far, single-level record printing seems to be working using this method for both the interpreted and compiled version.

Here's a sneak peak:

.type Car = [license:symbol, make:symbol, model:symbol]
.type Person = [name:symbol, id:number, car:Car]

.decl customer(person:Person, username:symbol, pin:number)
customer(Person["AZ", 67, Car["XYZ123", "Toyota", "Corolla"]], "azreika_souffle", 1234).

.decl result(person:Person, str:symbol)
result(Person[name, id, car], str) :-
    customer(Person[name, id, car], username, pin),
    str = cat("<< ", name, "(username: ", username, ", PIN: ", to_string(pin), ") >>").

.output result()

Currently have it printing:

result
person  str
===============
RECORD<2>: [0, 67, 1]   << AZ(username: azreika_souffle, PIN: 1234) >>
===============

Goal is to get it printing:

result
person  str
===============
Person[AZ, 67, Car["XYZ123", "Toyota", "Corolla"]]  << AZ(username: azreika_souffle, PIN: 1234) >>
===============

[josephcappadona]

Hi, given that it seems float support is still a ways away, what are my options (if any) for using floats and floating point arithmetic? The lack of such support makes working with Souffle a bit awkward for my use case.

[b-scholz]

Can you open a new issue asking how to have floats in the current version? I have a solution for you. Let's keep this thread related to the new type system.