Caching of provided types

[forki]

I'm working with @tforkmann on a project and we try to use SQLProvider. Unfortunately I don't have access to the db so the type provider can't generate types on my machine. Is there a way for @tforkmann to commit "cached" types so that I can build on my machine and look at the code?

[object]

Thanks for the PR. I will merge it and retest when I am back next week. I appreciate your help with design time functionality 'cause I don't have much experience with it.

When it comes to serializing stored procedures, there is an option of dropping DataContractJsonSerializer in favor of something else, but since this is about recursive F# types with functions, I am not sure if this can be easily resolved. Another option is to introduce a less complicated type to be used offline and serialize into it. Since the only thing that matters is making it compile, it might be doable.

[object]

I have merged a PR in my repo and left a comment about its behavior, but I believe before advancing we need to have a closer look at how to save the schema covering all queries in the app source code. While I was away, I met @sergey-tihon and told him about this idea. He found it too limiting if the schema is saved only for the queries listed in the current file or script, and he suggested to explore possibilities of saving schema in the end of the compilation. E.g.:

1. SaveContextSchema can still be defined and used as it's done now, for scenarios where it's easy to reference all referenced schema elements in a single source file.
2. During the solution compilation a type provider is instantiated, and it must build the sufficient schema for all queries referenced in the code.
3. The schema can be saved in the end of the compilation.

Now I must say that I lack the knowledge of how F# type providers are instantiated during the compilation phase and whether it's possible them to react on build events. But if this is possible, it opens for generation of schema based on all referenced entities.

/cc @Thorium

[Thorium]

He found it too limiting if the schema is saved only for the queries listed in the current file or script

What you mean by that? Doesn't it save the whole solution loaded to the editor? And you don't share SQLProvider context over different solutions anyway, there can be different file per solution.

[object]

Sorry for delay with response. I wanted to run more tests. In short: no, it doesn't save the schema for the whole solution. Here's how I tested.

1. I added 2 source files to the solution, both referring to the same SQL context type. Let's call them ModuleA and ModuleB.

2. ModuleA contained a function that ran a query from an Oracle SchemaA. Here's the query

   query { for p in ctx.SchemaA.MyTable do
           where (p.Media = "video") 
           select p.Description }

3. ModuleB contained a function that ran a query from an Oracle SchemaB.

   query { for p in ctx.SchemaB.MyTable do
           where (p.Media = "video") 
           select p.Description }

Both SchemaA and SchemaB (or "owner", I believe it's the same in Oracle) use the same connection string. The program compiled fine, at that time I didn't define ContextSchemaPath.

4. Then I defiend ContextSchemaPath and at the bottom of the ModuleA wrote the following:

sql.GetDataContext().SaveContextSchema().

The schema was saved according to the ContextSchemaPath.

5. But then if I try to compiled the solution, the compilation fails at ModuleB with the following errors:

error FS0039: The field, constructor or member 'Media' is not defined. error FS0039: The field, constructor or member 'Description' is not defined.

When I inspect the schema file, I can see that it saved definitions for all tables for all schemas, but only columns referenced in ModuleA where I put a call to SaveContextSchema.

6. If I duplicate the query from ModuleB in ModuleA and then save context schema, then the generated file is larger and compilation succeeds.

So there is a problem of schema cache applied only to the scope of the current file. Event though the whole solution compiles fine, the saved cache is restricted to columns referenced in the current file. Another observation: even though compilation fails, Intellisense works! If I hover a cursor over columns in file ModuleB, I see a tooltip with column names and types. Somehow there are two schema contexts generated, and the one that is saved is not generated for the whole solution.

[Thorium]

So they shared the same ctx?

Another observation: even though compilation fails, Intellisense works!

I was kind of hoping to save the intellisense-schema. Does this mean the cache dictionaries should be static totally and not per provided ctx? (Which could introduce new problems like cache not invalidating ever.)

Edit: The intellisense cache was probably just not cleared if you didn't close the whole editor after defining ContextSchemaPath.

[Thorium]

I don't know either how to save the schema at the end of compilation. Maybe ask from https://github.com/fsprojects/FSharp.TypeProviders.SDK ?

[object]

Yes, I was also hoping to save the Intellisense-schema, and in my understanding it should be the same as the one used by the compiler. Apparently it is not. I will investigate more.

[sergey-tihon]

what if you will go another way...

• Save provided typed (for offline use) only during compile time. (Using IsHostedExecution or IsInvalidationSupported from TypeProviderConfig to distinguish the case when compiler execute TP).
• If you cannot connect to the database you do not fail immediately but try to lookup types from a local file created during previous compilation.

[object]

@sergey-tihon This sounds interesting, but is there any way for an instantiated provider to detect that all provided type are created, so it's a right time to save the schema? Are there any callbacks that hits a provider about compilation stages?

[sergey-tihon]

is there any way for an instantiated provider to detect that all provided type is created, so it's a right time to save the schema?

I see... this is complicated because you do a lot of work lazily ... I do not know such callbacks that can help here.

The only solution that I see - updates the cache every time when compiler call TP

Instead of this

ty.AddMembersDelayed(builderFunc)

use

ty.AddMembersDelayed(builderFunc >> cacheFunc)

with some smart cache invalidation inside cacheFunc. But this path does not look easy ...

[object]

Thanks for the tip. Will give it a try.

[object]

Installed latest VS update, spent some various solutions, and in fact it all worked. Saving context schema produced a file with schema cache for the whole solution. So it looks promising.

[Thorium]

Merged. NuGet package 1.1.42.

[object]

Already received a question/suggestion from a co-worker whether design-time SaveContextSchema should be an optional measure: as long as ContextSchemaPath is set and the database is online, should the content of the file referenced by ContextSchemaPath be updated during the compilation? Doing this will ensure the schema cache is up-to-date. Something to think about.

[piaste]

That seems somewhat dangerous to me. It would be easy to miss the json file update among your commit changes.

If you want to occasionally compile against an online database, it's easy enough to comment out the static parameter line. You can even use a preprocessor flag to change it as a build configuration.

I'm still testing this new feature (I'm encountering runtime errors with it enabled, but I could be doing something wrong), but the way I would like to use it is to compile against MINIMUM_SUPPORTED_SCHEMA_VERSION even while my local database instance is being kept up-to-date with the newer changes. Automatically updating the schema file would not help in that case.

[object]

Agree, we have to be careful about supporting such implicit actions. The essence of F# type providers is that they generate types on the fly based on the actual data. Once there are multiple ways to bypass the live connections, things can become hard to control. I believe we should let the feature be used by different developers in different teams and then decide about improvements.

What kind of runtime error are you encountering? During the compilation or when the schema file is being generated?

[piaste]

@object

What kind of runtime error are you encountering? During the compilation or when the schema file is being generated?

Neither, both compilation and schema generation work fine. The error, strangely enough, is at runtime, it happens when the application calls .GetDataContext(runtimeConnString). It throws this exception:

The data contract type 'Microsoft.FSharp.Collections.FSharpMap`2[[System.String, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[FSharp.Data.Sql.Schema.Column, FSharp.Data.SqlProvider, Version=1.1.42.0, Culture=neutral, PublicKeyToken=null]]' cannot be deserialized because the required data members 'comparer, tree' were not found.

The line is here, it seems the DataContractSerializer for the .NET Core 2.0 runtime doesn't like the ColumnLookup type (Map<string, Column>), while the compilation runs on the full framework and doesn't throw.

After some fiddling, turns out the reason I was seeing it only at runtime is that when I run the application on my machine, SchemaCache.LoadOrEmpty finds the file and attempts to load it anyway. (If I manually rename the .json file before launching the application, the error doesn't occur.)

This part definitely needs fixing, as the library shouldn't try to load a random file path at runtime. It should be enough to always pass an empty contextSchemaPath to createProvider in SqlRuntime.DataContext.fs, I think?

After that, the serializer issue should only appear once the provider runs 100% on .NET Standard, including at design time. For that I'd support @Thorium 's idea and just go with JSON.Net like everybody else.

[keyset]

I'm running into this same problem. The type provider is locating the schema file and attempting to use it as a data source at runtime. I've filed issue #552 for this.

[ngriggsnd]

@object

Thank you for all your work on the SQLProvider.

I have been able to successfully use it to query data from an Oracle Database in an .fsx file.

However, I have not been able to get the SaveContextSchema(). function to work as I received the following compiler warning when hovering over the "dot" -> "Save failed: Access to the path 'C:\Users\nathaniel\Desktop' is denied"

I have tried running Visual Studio as the administrator but that had no effect.

Any guidance would be greatly appreciated. Thank you again.

Here is the code for reference [an .fsx file]:

r @"C:\Users\nathaniel\documents\visual studio 2015\Projects\Connect To Oracle\packages\SQLProvider.1.1.48\lib\net451\FSharp.Data.SqlProvider.dll"

open FSharp.Data.Sql

let [] Connection_String = @"User Id=hr;Password=password;Data Source=localhost" let [] Resolution_Path = @"C:\Users\nathaniel\documents\visual studio 2015\Projects\Connect To Oracle\packages\Oracle.ManagedDataAccess.12.2.1100\lib\net40" let [] Oracle_Owner = @"hr"

// C:\Users\nathaniel\Documents\Visual Studio 2015\Projects\Connect To Oracle\packages\Oracle.ManagedDataAccess.12.2.1100\readme.txt

type Oracle_SQL = SqlDataProvider< ConnectionString = Connection_String, DatabaseVendor = Common.DatabaseProviderTypes.ORACLE, ResolutionPath = Resolution_Path, Owner = Oracle_Owner, ContextSchemaPath = @"C:\Users\nathaniel\Desktop" , UseOptionTypes = true>

let database = Oracle_SQL.GetDataContext(SelectOperations.DatabaseSide) // limit data passed back and forth by chosing SelectOperation.DatabaseSide instead of DotNetSide

let result_of_query = query { for x in database.Hr.Employees do select ( x.Email ) } |> Seq.toArray

Oracle_SQL.GetDataContext(SelectOperations.DatabaseSide).SaveContextSchema().

[object]

@ngriggsnd "Save failed: Access to the path 'C:\Users\nathaniel\Desktop' is denied" doesn't necessarily mean that your app doesn't have rights to save the schema. It can also mean that the file is locked by another app (or even the same app). I don't know what it can be, so you need to investigate it yourself.

[ngriggsnd]

@object

Good news: I was able to save a schema file by changing "ContextSchemaPath = @"C:\Users\nathaniel\Desktop" to ContextSchemaPath = @"C:\Users\nathaniel\Desktop\meta.schema"

However, when I go to use the ContextSchemaPath static parameter in the .fs file, I am still receiving a compile time error/warning when trying to "dot" into the database. I am using the exact same query as in the .fsx file used to create the meta.schema file. So, from what I can tell, the lazy nature of the loading should not be an issue.

Please let me know if I am missing something or using the SQLProvider improperly. Thank you for your assistance.

[ngriggsnd]

@object

I was able to solve all compile-time issues by adding reference to System.Data as suggested and reported on this thread https://github.com/fsprojects/SQLProvider/issues/490 .

Please let me know if and how I might contribute to the documentation so that others may benefit.

[object]

@ngriggsnd That's good! Thanks for letting me know. I am not sure how the documentation should be updated for others to avoid this issue. My contribution to the project has been mainly around schema caching and Oracle provider, @Thorium should have better idea about whether System.Data reference should be mentioned. But you can browse current docs and if you have some suggestion about how to extend them, please go ahead and send a PR.

[dohly]

Hello @object ! SaveContextSchema() doesn't work for me. I'm using SQL server express. Everything works well (ActivityName property is available) if ContextSchemaPath is not specified. Otherwise it doesn't work (no columns\properties available). I see that schema file was created. My version of SQLProvider is 1.1.49 Please help. Thanks.

UPD: I've just found solution - downgrade to 1.1.48

[Thorium]

@dohly: The schema-files are not compatible between 1.1.48 and 1.1.49. So if you generated schema file with 1.1.48, you have to re-genarete it for 1.1.49. The reason is that 1.1.49 added serialization of stored procedures.

[object]

Oh that's good to know. Was about to upgrade SQLProvider myself :-)