Errors running own-compiled ApsimX

[jal-frezie]

Having compiled ApsimX from source in Debian Buster, I get this message when I open a model:

System.TypeLoadException: Could not load type of field 'UserInterface.Views.TWWebBrowserWK:Browser' (0) due to: Could not load file or assembly 'webkit-sharp, Version=1.1.15.0, Culture=neutral, PublicKeyToken=eaa1d335d2e19745' or one of its dependencies. at UserInterface.Views.HTMLView.PopulateView (System.String contents, System.Boolean isURI) [0x000cb] in <26124421fb0f419cae2f222cd96186ab>:0 at UserInterface.Views.HTMLView.SetContents (System.String contents, System.Boolean allowModification, System.Boolean isURI) [0x00026] in <26124421fb0f419cae2f222cd96186ab>:0 at UserInterface.Presenters.GenericPresenter.Attach (System.Object model, System.Object view, UserInterface.Presenters.ExplorerPresenter explorerPresenter) [0x000ec] in <26124421fb0f419cae2f222cd96186ab>:0 at UserInterface.Presenters.ExplorerPresenter.ShowInRightHandPanel (System.Object model, System.String viewName, System.String presenterName) [0x00071] in <26124421fb0f419cae2f222cd96186ab>:0

The model seems to open and run OK. I get similar messages when running it. However when I try to add a new manager script to my model I get this error:

System.Reflection.ReflectionTypeLoadException: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. Could not load type of field 'UserInterface.Views.TWWebBrowserWK:Browser' (0) due to: Could not load file or assembly 'webkit-sharp, Version=1.1.15.0, Culture=neutral, PublicKeyToken=eaa1d335d2e19745' or one of its dependencies. at (wrapper managed-to-native) System.Reflection.Assembly.GetTypes(System.Reflection.Assembly,bool) at System.Reflection.Assembly.GetTypes () [0x00000] in <7b90a8780ac4414295b539b19eea7eea>:0 at APSIM.Shared.Utilities.ReflectionUtilities.GetTypesThatHaveInterface (System.Type interfaceType) [0x0001d] in <3e558596337e4479a6739a574b89f693>:0 at Models.Core.Apsim.GetAllowableChildModels (System.Object parent) [0x00008] in :0 at UserInterface.Presenters.AddModelPresenter.Attach (System.Object model, System.Object view, UserInterface.Presenters.ExplorerPresenter explorerPresenter) [0x00026] in <26124421fb0f419cae2f222cd96186ab>:0 at UserInterface.Presenters.ExplorerPresenter.ShowInRightHandPanel (System.Object model, System.String viewName, System.String presenterName) [0x00071] in <26124421fb0f419cae2f222cd96186ab>:0 System.TypeLoadException: Could not load type of field 'UserInterface.Views.TWWebBrowserWK:Browser' (0) due to: Could not load file or assembly 'webkit-sharp, Version=1.1.15.0, Culture=neutral, PublicKeyToken=eaa1d335d2e19745' or one of its dependencies.

... I cannot add any component to the model. Noting the mentions of webkit-sharp in the above, I checked for matching installed packages:

$ apt list webkitsharp* Listing... Done libwebkit2-sharp-4.0-cil-dev/stable,stable,now 2.10.9+git20160917-1.1 amd64 [installed] libwebkit2-sharp-4.0-cil-dev/stable,stable 2.10.9+git20160917-1.1 i386 libwebkit2-sharp-4.0-cil/stable,stable,now 2.10.9+git20160917-1.1 amd64 [installed] libwebkit2-sharp-4.0-cil/stable,stable 2.10.9+git20160917-1.1 i386

...which seems to indicate I have what is required. If I am using the .deb package downloaded from apsim.info, everything works OK. Unfortunately I cannot open my models with that version now because it says they have been opened with a newer version!

Any help?

[hol430]

Hi @jal-frezie, can you try copying webkit-sharp.dll and webkit-sharp.dll.config from ApsimX/ApsimNG/Assemblies into ApsimX/Bin/?

I suspect the real problem here is that the instructions on the website are out of date.

[zur003]

A few things to try: 1) Make sure that you have libwebkit1.1-cil installed. 2) Make sure that you have a copy webkit-sharp.dll.config in the same folder as webkit-sharp.dll. 3) If you look inside the webkit-sharp.dll.config file, you will see that it points to libwebkitgtk-1.0.so.0. This file needs to be installed on your system where the loader can find it. For me, it's in /usr/lib/x86_64-linux-gnu/ - check to be sure this is findable on your system.

[zur003]

@hol430 - Looking at this, I wonder whether a number of things are out-of-date. The instructions (and my earlier advice) tell the user to install libwebkit1.1-cil, but for Ubuntu that package apparently hasn't been available since Xenial (16.04). We made need to re-evaluate how we load and access Webkit from ApsimX. I'm not sure how much of this nuget can handle.

[zur003]

Try installing libwebkitgtk-1.0-0.

Also let us know if any of this solves the issue.

@hol430 - What are your thoughts on supporting this in the longer term? Ubuntu (and I suppose Debian generally) seem to be gradually withdrawing support for libwebkit1 under gtk2 in favour of libwebkit2 under gtk3 (I think). Should we be looking at transitioning to gtk3? This might be better overall, as I think there are nuget packages for it. Not sure whether it supports both 32-bit and 64-bit native layers.

[jal-frezie]

Following Drew's instructions above fixed the problem.

[hol430]

In the long run there should be some advantages in using gtk3 - it's supposed to perform better, styling will be easier (I've always had trouble with the .rc files gtk2 uses), more widgets, bugfixes, easier package management, etc. From what I can gather, webkit-sharp is being dropped from most distros due to unpatched security vulnerabilities. The package maintainers seem to suggest moving to webkit2-sharp (which also implies a move to gtk3/webkit2) so that's another good reason.

I guess it depends on how much work would be involved in transitioning from gtk2 to 3. There is a short guide on the mono website, but I'm unsure if it's short because the process is simple, or short because they haven't updated it.

Also - is there a lot of demand for 32-bit apsim next gen?

[zur003]

I just had a look at some of my correspondence from 3 years ago, trying to refresh my memory about why I went with Gtk2 rather than Gtk3 at the time. Here's what I wrote:

Why Gtk2 rather than Gtk3? Mainly because the Mono people have stuck with it; when you install Mono, the Gtk# package that comes with it is based on Gtk2. Their help library describes Gtk2, not Gtk3, and the Mono.TextEditor is based on Gtk2. Also, when I was first playing around with Gtk3, I had trouble getting bits of the Windows implementation to work correctly. If I recall, I wasn’t seeing a lot of the icons, and things seemed a bit unstable. Now that I understand the overall Gtk architecture a bit better, I could possibly work through most of those issues, but at the time stability seemed more important than feature-richness. Also, some of the design of Gtk3 makes it a bit more awkward to use with individual applications – it was designed for use with the Gnome 3 desktop, so much of the theming and other configuration is done more easily on a system level, rather than application level.

My views about theming at the time were largely based on my ignorance, and that shouldn't be an issue. The bigger problem might be the Mono.TextEditor component. At the time, there weren't a lot of options for embeddable platform-independent code editors in Gtk# - I expect there still aren't.

That said, though, I think we ought to start looking at moving the GUI to Gtk3. And I think at this point we could probably drop support for 32-bit.

[zur003]

@hol430 , @hol353 - I've had a look at what would be involved in moving to Gtk3. As I anticipated, the biggest challenge appears to be the Mono.TextEditor component, where a number of low-level operations would need to be done differently. Our own code would also need quite a few changes, as Widget styles are handled differently (there is no longer a Widget.Style property, for example). I estimate it would take around a month to get it all working. So it's doable, but far from trivial.

So what do you think?

[zur003]

Ah, and it also appears that getting webkit2-sharp working nicely might take some effort as well. The pre-compiled version is looking for a gtk-sharp.dll, but the nuget packages I used for Gtk3 call it GtkSharp.dll. There may be other problems of this sort.

[hol353]

Ok @zur003 . Let's have a go at migrating to gtk3. It would be nice if everything was in NuGet. Let's also drop support for 32 bit.

[zur003]

@hol353, @hol430 - Or maybe we should get even more radical, and try porting to Qml.Net (https://qmlnet.github.io/). I have no idea how feasible or infeasible that might be. Qt offers a very rich set of controls, but possibly a lot of overheads...

[hol353]

Hmm. I would only consider this if there were some clear advantages e.g. if Qt offers a better grid control I might think about it.

[lie112]

Any thoughts on moving to .Net Core for better cross platform compatibility in future. The speed improvements are meant to be good and will provide many benefits. It would also mean we are no longer tied to an old .Net framework version and can get the latest C# coding practices. There is GTK# 3 for Core.

[hol353]

Yep I agree! Let's do it.

[hol430]

Yeah I think there was some discussion with @jbrider in an old pull request (can't remember which!) about this where we agreed it would be a bit of work to get it working. I agree in the long run we're almost certainly better off running on dotnet core. Maybe it makes sense to move to dotnet core at the same time we move to gtk3? I guess that's starting to become a pretty big job though.

[jbrider]

.net core 3 is being released this month, so the timing is good. We had a couple of dependencies that weren't compatible with .net core, but more than 90% of our codebase was compatible so I'm don;t think it's a big issue - more one of finding the time to make it happen I think the gtk3 may be the harder part.

On Tue, Sep 24, 2019 at 10:01 AM Drew Holzworth notifications@github.com wrote:

Yeah I think there was some discussion with @jbrider https://github.com/jbrider in an old pull request (can't remember which!) about this where we agreed it would be a bit of work to get it working. I agree in the long run we're almost certainly better off running on dotnet core. Maybe it makes sense to move to dotnet core at the same time we move to gtk3? I guess that's starting to become a pretty big job though.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/APSIMInitiative/ApsimX/issues/4160?email_source=notifications&email_token=AAE3G7GROVRCC4VEVYZRCL3QLFKHLA5CNFSM4IUAVWMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7MT4NA#issuecomment-534330932, or mute the thread https://github.com/notifications/unsubscribe-auth/AAE3G7AYGDSXQTOXEW2CXL3QLFKHLANCNFSM4IUAVWMA .

[zur003]

It appears that .net core 3.0 was released today. I've had a quick play with it, and used it to get a very basic "Hello, world" GtkSharp 3 window to display on Windows and Ubuntu Linux. I suspect @jbrider is correct - the hardest part will be moving to Gtk3. Our user interface is actually pretty sophisticated, with a code editor, charting, embedded HTML viewer, etc. Parts of it will port fairly easily, but I'm sure that some parts won't.

I still wonder whether Qt (via Qml.NET) is worth considering for the GUI. I think it has a proper spreadsheet widget, and a lot of other nice features, and it does work with .net core. At this point I don't really know enough about the architecture to really give a proper evaluation, though. It would be interesting to see how hard/easy it is to create installation packages, and how easily it ports across platforms.

[hol353]

Well, if it has a better grid control then perhaps it is worth doing an evaluation for a week or so and come back with a list of pros and cons.

[jbrider]

I did a quick scan using a couple of tools and found the following list of libraries that aren't supported by dotnet core. There is definitely a dotnet core version of EntityFramework - which should mean there are some SQLite libraries as well, or not far away. Some of the others may take some work to work around.

ICSharpCode.NRefactory ICSharpCode.TextEditor Mono.Cecil MarkdownDeep.NET OxyPlot.GtkSharp OxyPlot.Core PDFsharp-gdi PDFsharp-MigraDoc EntityFramework System.Data.SQLite.EF6 System.Data.SQLite.Linq

On Tue, Sep 24, 2019 at 1:53 PM Dean Holzworth notifications@github.com wrote:

Well, if it has a better grid control then perhaps it is worth doing an evaluation for a week or so and come back with a list of pros and cons.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/APSIMInitiative/ApsimX/issues/4160?email_source=notifications&email_token=AAE3G7D4VYD4KZCUUHLQGQDQLGFKJA5CNFSM4IUAVWMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7M7EUA#issuecomment-534377040, or mute the thread https://github.com/notifications/unsubscribe-auth/AAE3G7F7SVN6PZXKPGISP43QLGFKJANCNFSM4IUAVWMA .

[hol430]

We should in theory be able to replace ICSharpCode.NRefactory with roslyn, which I think runs on .net core. This should also eliminate the need for Mono.Cecil as I think it's only used by the intellisense.

[jbrider]

I found an article on using PDFsharp MigraDoc with dotnet core. Not straight forward, but looks like it would work.

https://damienbod.com/2018/09/30/creating-pdf-files-in-asp-net-core/

On Wed, Sep 25, 2019 at 8:33 AM Drew Holzworth notifications@github.com wrote:

We should in theory be able to replace ICSharpCode.NRefactory with roslyn, which I think runs on .net core. This should also eliminate the need for Mono.Cecil as I think it's only used by the intellisense.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/APSIMInitiative/ApsimX/issues/4160?email_source=notifications&email_token=AAE3G7A3Y4HEXUH7SZVW6LDQLKITBA5CNFSM4IUAVWMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7QAFII#issuecomment-534774433, or mute the thread https://github.com/notifications/unsubscribe-auth/AAE3G7ETVXZMQIMGOBTTI3LQLKITBANCNFSM4IUAVWMA .

[zur003]

@jbrider, @hol353 @hol430 - I took a few hours today to play with my newly installed .Net core 3, and have successfully built APSIM.Shared and Models.exe using it. The hardest part was working out how to get Roslyn to compile manager scripts at runtime (that bit still needs some work). On Windows, execution time seems to be very close to the same as the .Net Framework 4.6 version. I haven't tried it yet on non-Windows platforms, but in theory it should be faster than Mono. There appear to be a few little quirks. For example, the datastore didn't close properly, leaving .db-shm and .db-wal files behind.

Just for the record, at this point the only code changes that were needed were those for script compilation. Otherwise, all that was needed was tracking down appropriate NuGet packages.

I've also taken a little closer look at Qml.NET, and don't think it would be worth the very substantial effort that would be required to go that direction. My view is that Gtk# - either 2 or 3 - remains our best option for a cross-platform GUI, despite its deficiencies.

[hol353]

Thanks for the update. Is it worth migrating to Gtk#3?

[zur003]

I'm not sure about the migration to Gtk#3 - that will take a bit more effort, for possibly less reward. Porting to .Net Core 3 may not seem like a great step forward, but execution times on Linux are much faster than when using Mono (not much difference at all on Windows, though). Getting the GUI working will probably take some time, but Models.exe works now, and could be very useful at speeding up things like Docker runs. If anyone else wants to play with it, I've added a netcore3 branch in the git repository.

[hol353]

Ok lets migrate to .Net Core 3.

[jbrider]

Nice work Eric.

On Fri, Sep 27, 2019 at 1:58 PM Dean Holzworth notifications@github.com wrote:

Ok lets migrate to .Net Core 3.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/APSIMInitiative/ApsimX/issues/4160?email_source=notifications&email_token=AAE3G7ESJCXSLZBLMUMREJTQLWAGVA5CNFSM4IUAVWMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7XUV5Q#issuecomment-535775990, or mute the thread https://github.com/notifications/unsubscribe-auth/AAE3G7AFMJJ664ECGJQFEJLQLWAGVANCNFSM4IUAVWMA .

[zur003]

The hard part is going to be the GUI. The GtkSharp NuGet packages for .net core target Gtk#3; we could stick with Gtk#2, but would probably need to compile the various DLLs ourselves. Fortunately pretty much everything is open source, so that should be possible. I'm not sure whether the best way forward is to first try getting Gtk#2 running on .Net core, or jumping straight in to Gtk#3. I suspect going to Gtk#3 would be better, but will probably require a fair bit of effort with the embedded code editor and web browser.

[hol430]

Does webkit2-sharp work on Windows/Mac? The html view will be much more pleasant to work with if we use the same browser widget across all operating systems.

[hol430]

@zur003 have you looked into the SourceView at all for a code editor? From the documentation it looks like it's shipped with gtk#, although it must be in its own assembly because we don't seem to have access to it currently.

[zur003]

I've been playing around some more with .Net core 3 and GtkSharp 3 for the last few days, and thought I should report back on how it all looks. As I mentioned last week, getting the core non-GUI stuff working under .Net Core was fairly easy. That all seems to run OK, is much faster than Mono on Linux, and slightly faster that .Net Framework on Windows. The GUI, though, is not so simple. I've been able to get things running with Gtk#2 under .Net Core on Windows, although that required rebuilding Gtk-sharp and the Mono.TextEditor, and replacing some of the NuGet packages with alternatives. Not everything is working - I had to disable all the code completion stuff, as ICSharpCode.NRefactory hasn't been (and won't be) ported to .Net core. The correct way to do it would be to make use of the code completion support in the Roslyn compiler (see https://www.strathweb.com/2018/12/using-roslyn-c-completion-service-programmatically/ ). Web browser support is a bit of a headache - currently on Windows this works through System.Windows.Forms providing access to the MS WebBrowser control. On .Net core, pulling in System.Windows.Forms seems to break execution on non-Windows platforms, so some alternative is needed. It would be nice if the embedded browsing would work across platforms. There is a commercial product, EO.WebBrowser link, which looks like it would suit our needs, but would we want to purchase and use a non-open source component?

Porting from GtkSharp2 to GtkSharp3 also has its challenges. The support for core GtkSharp functionality should be a bit easier, as there are NuGet packages for this. However, the Mono.TextEditor component would need some modification.

I do wonder whether this is all worth the trouble, although the speed gains of .Net core would be good to have, and it appears that Microsoft plans to be moving everything in that direction.

[hol430]

Which version of gtk-sharp are you using? There seem to be two main repos on github - a gtk2 version owned by mono and a gtk3 version (which I think is supposed to work with .net core).

[zur003]

For the moment, I'm building Gtk# from source. I've been playing around with both gtk2 and gtk3 on .Net core. Each has its own problems. The problem with Gtk3 is that I haven't been able to get Mono.TextEditor to work with it. I get a bit further with Gtk2, and have build a GUI I can run on Windows, but haven't been able to run it on Linux. Mono has a mechanism that uses .config files to tell an application where to locate needed native binaries; for example, that when webkit-1.0 is requested, it should map that request to libwebkitgtk-1.0.so.0. Things don't work that way with .Net core. I'm not really sure how it does work, and don't yet know how to tell the application where to find the native gtk+2 .so files.

For what it's worth, I've had a quick look at the commercial EO.WebBrowser for .NET, and it looks promising as a way to simplify cross-platform browser support. The question is whether we would want to include a commercial product in our open-source project.

[hol430]

What I meant was where are you getting the gtk# source code from?

Wrt the text editor - I haven't looked into this but there's a source code text editor called SourceView which is supposed to ship with gtk#. No idea how good it is though - I believe monodevelop was previously using this editor and decided to drop it in favour of their own Mono.TextEditor, so maybe it has its problems.

Wrt to the browser - does webkit (or the newer webkit2) work on windows/mac?

[hol430]

Closing this as the original issue is fixed.