This is a big PR that I've been looking forward to, now unblocked by #5380.
The goals of this PR are that:
name-resolution is 100% done at parse-time (never at run-time)
package items (types, consts, fns) are referenced, in the AST, by ID, rather than by name
(this removes ambiguities, sets us up for by-hash references, and generally 100%-predictable code, driven by the PM)
(really, this is prep-work for even more Package Manager work)
Towards those goals, the following changes have been made:
in ProgramTypes (and SerializedTypes, and others down the line), FQTypeName.Package is now just a uuid
relatedly, PackageType.Name now exists, with the { owner, modules, name } bits
we now have 2 Package Manager concepts - a Run-Time one and a (Program-/Parse)-Time one
the program-time one:
has findFn-type functions, which take a PackageFn.Name and return just an ID (if found)
has getFn-type functions, which take just ID and return the data
the run-time one only has getFn-type functions
(there's never a need to "look up a type by name" at run-time, except for error-reporting, which is dealt with in Darklang, via the PM)
in the many places in our F# codebase where we previously referenced package items by name, such as in Builtins, RTEs, etc, we now have to reference them by ID.
so, all items that we reference from F# are now consolidated in a single file, LibExecution/PackageIDs.
this module^ mananages pre-set UUIDs, per named Package Type/Fn (and future, const) that we reference from F#.
along with the IDs set per referenced thing, we also note the name
at parse-time, if we stumble upon one of those named things (like type Stldib.Option.Option), we use the corresponding ID when saving to the DB, so that the setup works
this accounts for a large # of the "files changed" in this PR
previously, we had a PackageManager module written in Darklang, which directly interfaced with the endpoints exposed by the dark-packages canvas
given that we use this a lot more now (to pretty-print RTEs at run-time), we need to have reasonable performance
so, we now expose a number of Builtin functions for looking up and fetching package items
these Builtins are based on the Program-time Package manager, not the run-time one
in the Cloud runtime, this directly accesses our SQL DB
in the CLI runtime, we use the LibPackageManager project to access the HTTP endpoints
(we support this in a similar way to how we injected a 'config' into the HttpClient builtin module)
both versions deal with caching
naturally, the package fns in the PackageManager module are wrappers around these newly-exposed builtins
many RuntimeErrors previously relied on the toStringing of FQTypeName and the like, which... no longer has actual name bits, but now just has UUIDs, for package items
so, at least in all tested RTEs, I've gone away from OldErrorMessageTODO, to proper typed RTEs
and the stringification is now in Darklang
(a few error messages improved as part of this)
pretty-printing, in Darklang, had to adjust for the FQ[x]Name/Package[x].Name changes
now, we have to look up the names, by ID
because of this, the old roundtripping tests in testfiles/../parser.dark got broken
we parse the declarations, and immediately try to print them
but, the declarations haven't been persisted, so when we go to pretty-print them,
the package manager can't find the items to pretty-print, and we fail
(actually we currently don't fail: these end up in NameResolutionError.NotFounds, and we print those plainly - but we should update this to be truer roundtripping tests)
so, those roundtripping tests are now in F# -- see NewParser.Tests.fs
I probably need to follow up in NewParser.Tests.fs and assess more. See the big TODO in there, and at the bottom of this PR description.
Minor changes along the way:
in the DB, renamed package_types_v0 column typename to just name, and similarly for fns
the IntRuntimeError module was previously hiding in Builtins/Int64.fs, and referenced by all the other Int-type builtin modules - pulled out to its own file
for consistency, type PackageConstant lives in a PackageConstant wrapper module; same for PackageType
I re-shaped NameResolutionError a bit
the NameResolver in Darklang is much nicer (even better than last wave), and more in-line with the F# equivalent
some more types previously named T now match their parent module's name (for consistency w/ equivalent Dark types)
in Serialization.TestValues.fs, I settled on using a single consistent id for all of the things - it's too tiring to try to use different IDs, and didn't really help with anything
the dark-packages endpoints now feel notably snappier
Some follow-ups:
[x] consider removing RT.PackageType.Name - I don't think we actually need it any more
[ ] continue to type-ify more Runtime Errors; remove usages of FQ[x]Name.toString so we don't show (only) UUIDs in error messages and such
[ ] revisit NameResolutionError, and "RTEs" more broadly - some of these (i.e. NRE.ErrorType.InvalidName) only really happen at parse-time
[ ] revisit NewParser.Tests.fs - there's a big TODO in there that needs thinking about. Maybe we can go back to having these tests in testfiles/../parser.dark, or maybe we need some adjustment there to disallow unresolved names.
This is a big PR that I've been looking forward to, now unblocked by #5380.
The goals of this PR are that:
Towards those goals, the following changes have been made:
ProgramTypes(andSerializedTypes, and others down the line),FQTypeName.Packageis now just auuidPackageType.Namenow exists, with the{ owner, modules, name }bitsfindFn-type functions, which take aPackageFn.Nameand return just anID(if found)getFn-type functions, which take justIDand return the datagetFn-type functionsLibExecution/PackageIDs.type Stldib.Option.Option), we use the corresponding ID when saving to the DB, so that the setup worksPackageManagermodule written in Darklang, which directly interfaced with the endpoints exposed by thedark-packagescanvasLibPackageManagerproject to access the HTTP endpointsHttpClientbuiltin module)PackageManagermodule are wrappers around these newly-exposed builtinstoStringing ofFQTypeNameand the like, which... no longer has actual name bits, but now just has UUIDs, for package itemsOldErrorMessageTODO, to proper typed RTEsFQ[x]Name/Package[x].Namechangestestfiles/../parser.darkgot brokenNameResolutionError.NotFounds, and we print those plainly - but we should update this to be truer roundtripping tests)NewParser.Tests.fsNewParser.Tests.fsand assess more. See the big TODO in there, and at the bottom of this PR description.Minor changes along the way:
package_types_v0columntypenameto justname, and similarly for fnsIntRuntimeErrormodule was previously hiding inBuiltins/Int64.fs, and referenced by all the other Int-type builtin modules - pulled out to its own filetype PackageConstantlives in aPackageConstantwrapper module; same forPackageTypeNameResolutionErrora bitTnow match their parent module's name (for consistency w/ equivalent Dark types)Serialization.TestValues.fs, I settled on using a single consistentidfor all of the things - it's too tiring to try to use different IDs, and didn't really help with anythingdark-packagesendpoints now feel notably snappierSome follow-ups:
RT.PackageType.Name- I don't think we actually need it any moreFQ[x]Name.toStringso we don't show (only) UUIDs in error messages and suchNameResolutionError, and "RTEs" more broadly - some of these (i.e.NRE.ErrorType.InvalidName) only really happen at parse-timeNewParser.Tests.fs- there's a big TODO in there that needs thinking about. Maybe we can go back to having these tests intestfiles/../parser.dark, or maybe we need some adjustment there to disallow unresolved names.