Missing imports sometimes result in very confusing error trace

[radeusgd]

The mentioned bug seems a bit random and very hard to track down, I did not manage to find a small self-contained repro.

But it's been causing problems to me and @jdunkerley quite a lot in the recent weeks, in quite random moments.

The issue is that sometimes when running the project, we get a wall of errors saying that some names/types could not be found, for example:

In module Standard.Database.Internal.Redshift.Redshift_Dialect:
Compiler encountered errors:
Redshift_Dialect.enso[29:28-29:43]: The name `Postgres_Dialect` could not be found.
Redshift_Dialect.enso[42:12-42:15]: The name `Text` could not be found.
Redshift_Dialect.enso[51:20-51:24]: The name `Query` could not be found.
Redshift_Dialect.enso[51:29-51:41]: The name `SQL_Statement` could not be found.
Redshift_Dialect.enso[53:9-53:22]: The name `Base_Generator` could not be found.
Redshift_Dialect.enso[67:32-67:46]: The name `Internal_Column` could not be found.
Redshift_Dialect.enso[67:51-67:64]: The name `Sort_Direction` could not be found.
Redshift_Dialect.enso[67:69-67:75]: The name `Nothing` could not be found.
Redshift_Dialect.enso[67:79-67:91]: The name `Text_Ordering` could not be found.
Redshift_Dialect.enso[67:96-67:111]: The name `Order_Descriptor` could not be found.
Redshift_Dialect.enso[69:9-69:24]: The name `Postgres_Dialect` could not be found.
Redshift_Dialect.enso[74:20-74:29]: The name `Connection` could not be found.
Redshift_Dialect.enso[74:34-74:46]: The name `SQL_Join_Kind` could not be found.
Redshift_Dialect.enso[74:51-74:54]: The name `Text` could not be found.
Redshift_Dialect.enso[74:59-74:67]: The name `From_Spec` could not be found.
Redshift_Dialect.enso[74:72-74:80]: The name `From_Spec` could not be found.
Redshift_Dialect.enso[74:85-74:90]: The name `Vector` could not be found.
Redshift_Dialect.enso[74:95-74:100]: The name `Vector` could not be found.
Redshift_Dialect.enso[74:105-74:110]: The name `Vector` could not be found.
Redshift_Dialect.enso[74:115-74:119]: The name `Table` could not be found.
Redshift_Dialect.enso[76:9-76:28]: The name `Database_Join_Helper` could not be found.
Redshift_Dialect.enso[81:27-81:30]: The name `Text` could not be found.
Redshift_Dialect.enso[81:35-81:41]: The name `Boolean` could not be found.
Redshift_Dialect.enso[88:26-88:39]: The name `Naming_Helpers` could not be found.
Redshift_Dialect.enso[89:31-89:44]: The name `Naming_Helpers` could not be found.
Redshift_Dialect.enso[94:24-94:39]: The name `SQL_Type_Mapping` could not be found.
Redshift_Dialect.enso[95:29-95:49]: The name `Postgres_Type_Mapping` could not be found.
Redshift_Dialect.enso[100:36-100:43]: The name `SQL_Type` could not be found.
Redshift_Dialect.enso[100:48-100:61]: The name `Column_Fetcher` could not be found.
Redshift_Dialect.enso[104:9-104:29]: The name `Column_Fetcher_Module` could not be found.
Redshift_Dialect.enso[107:28-107:43]: The name `Statement_Setter` could not be found.
Redshift_Dialect.enso[108:33-108:48]: The name `Postgres_Dialect` could not be found.
Redshift_Dialect.enso[111:31-111:46]: The name `Aggregate_Column` could not be found.
Redshift_Dialect.enso[111:51-111:57]: The name `Boolean` could not be found.
Redshift_Dialect.enso[111:61-111:90]: The name `Unsupported_Database_Operation` could not be found.
Redshift_Dialect.enso[114:9-114:12]: The name `True` could not be found.
In module Standard.Database.Connection.Database:
Compiler encountered errors:
Database.enso[18:12-18:27]: The name `Postgres_Options` could not be found.
Database.enso[18:29-18:42]: The name `SQLite_Options` could not be found.
Database.enso[18:44-18:59]: The name `Redshift_Options` could not be found.
Database.enso[18:65-18:82]: The name `Connection_Options` could not be found.
Database.enso[18:87-18:96]: The name `Connection` could not be found.
Database.enso[18:100-18:108]: The name `SQL_Error` could not be found.
Database.enso[19:25-19:42]: The name `Connection_Options` could not be found.
Aborting due to 43 errors and 1 warnings.

But this often refers to pretty standard types like Text and Nothing which surely are imported in this file!

To reproduce it:

• please checkout the branch wip/radeusgd/checkpoint-import-bug-repro,
• run sbt buildEngineDistribution
• run tests\Table_Tests\src\Database\Commpon_Spec.enso.

Other repros are quite possible, but this bug is very brittle and random. The steps above are ones that allowed me to reproduce the issue consistently.

[radeusgd]

Apparently, the following patch seems to fix the issue that is present on the linked branch:

Index: distribution/lib/Standard/Database/0.0.0-dev/src/Internal/Redshift/Redshift_Dialect.enso
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/distribution/lib/Standard/Database/0.0.0-dev/src/Internal/Redshift/Redshift_Dialect.enso b/distribution/lib/Standard/Database/0.0.0-dev/src/Internal/Redshift/Redshift_Dialect.enso
--- a/distribution/lib/Standard/Database/0.0.0-dev/src/Internal/Redshift/Redshift_Dialect.enso  (revision f6f7aa541f6d7e47c8af6c7a1760bd28e383f22a)
+++ b/distribution/lib/Standard/Database/0.0.0-dev/src/Internal/Redshift/Redshift_Dialect.enso  (date 1679937196832)
@@ -17,8 +17,9 @@
 import project.Internal.IR.Query.Query
 import project.Internal.Postgres.Postgres_Dialect
 import project.Internal.Common.Database_Join_Helper
-import project.Internal.SQL_Type_Mapping.SQL_Type_Mapping
 import project.Internal.Postgres.Postgres_Type_Mapping.Postgres_Type_Mapping
+import project.Internal.SQL_Type_Mapping.SQL_Type_Mapping
+import project.Internal.Statement_Setter.Statement_Setter
 from project.Errors import Unsupported_Database_Operation

 ## PRIVATE

So it seems the issue has something to do with imports.

Applying this patch and rebuiliding with buildEngineDistribution makes stuff compile. The test suite runs. It fails because I took a snapshot of my branch at a point where a test was failing. But the indication that stuff changed is that the tests actually get to run.

Once I revert this mentioned patch, and rebuild again with buildEngineDistribution, I again get the wall of text of 'not found' errors, as above.

[radeusgd]

@jdunkerley @hubertp could you confirm if you can repro the issue using the steps I listed above?

It is very brittle so I want to check if I provided all necessary details. I manage to consistently reproduce it on my machine, but I'm not sure if that is enough.

[hubertp]

@radeusgd can't reproduce w/ or w/o caches using the provided instructions.

[hubertp]

Also, if I was to bet, I suspect it could have something to do with outdated caches of IRs/bindings.

[hubertp]

I was eventually able to confirm. And yes, it seems to be cache-related. If there is an error when generating the local caches, like missing import, it seems to be silent. As a result, having a partial cache and an existing source code, leads to rather obscure error message. First, investigating if we can report problems much quicker, when generating the index. Second, will need to see if we can somehow work with such half-broken caches and bindings.

[enso-bot[bot]]

Hubert Plociniczak reports a new STANDUP for yesterday (2023-03-28):

Progress: More investigation into reported issues. Added more debug info 6099 in the absence of steps to reproduce. Struggled to reproduce 6092 but eventually got it. One problem is that we silently dropping errors during cache generation. The other is that we should be able to recover from caching issues. It should be finished by 2023-03-29.

Next Day: Next day I will be working on the #6092 task. Provide a workaround for reporting errors. Catch up on the final design of execution contexts.

[enso-bot[bot]]

Hubert Plociniczak reports a new STANDUP for today (2023-03-29):

Progress: Added a workaround for reporting errors. Started to look into caches when dealing with broken caches. Filed 6135 as a follow up, since it will take a bit more time to get right and it is not the most pressing issue anymore. Meetings on execution contexts, scheduled tickets with follow up work. Started working on 6129 by replacing IOEnvironment with contexts. It should be finished by 2023-03-29.

Next Day: Next day I will be working on the #6129 task. Continue implementing the changes according to the design.