search

DHI / mikeio

Read, write and manipulate dfs0, dfs1, dfs2, dfs3, dfsu and mesh files.
https://dhi.github.io/mikeio
BSD 3-Clause "New" or "Revised" License
139 stars 56 forks source link

Any plans for a Linux enabled version? #50

Closed bhlevca closed 3 years ago

bhlevca commented 4 years ago

The current version in __init__.py is loading dlls. If the equivalent functionality can be replaced with shared libraries perhaps a Linux or MacOS version could be achieved.

It is possible that the Linux version could be made available sooner as there are already binaries for the cloud enabled Linux version of Mike Zero.

bhlevca commented 4 years ago

If I am given some guidance I can try to implement this.

halvgaard commented 4 years ago

@bhlevca Great :) We will hit summer break soon so might not be able to help much in July. If you manage to setup a quick draft, you can PR your branch into master as a draft that we can follow and discuss and link to this issue.

bhlevca commented 4 years ago

@rhaDHI Thanks for your answer. What kind of draft you have in mind?
I will need some information about the Linux shared libraries that contain the information needed by mikeio to handle the DFS files.
Also, I see there is a dependency on the .Net API, which I suppose is used in the Mike DLLs. Linux has some issues with that, but Mono development tools may help with that.

halvgaard commented 4 years ago

@bhlevca I think that @JesperGr mentioned that it is possible without Mono.

bhlevca commented 4 years ago

@rhaDHI, @JesperGr, that would be great. If I get some starting tips, I could try working on it.

bhlevca commented 4 years ago

@rhaDHI I am ready to try my hand at it. You told me above that it would be possible to do it without Mono. If I get a push in the right direction I will try to develop the glue code for this.

halvgaard commented 4 years ago

@JesperGr Any suggestions?

JesperGr commented 4 years ago

Right now the library is based on the MIKE Core .NET libraries, and utilizing the ease of the .NET API. That requires .NET integration in python.

I have no idea if the .NET integration in python works on Linux, or whether it can be made to work. I guess that would be the first thing to test - testing need not be based on MIKE Core assemblies, but just getting any .NET Standard library working in python could show if that is an option.

The alternative is to use the C API to DFS files. Using a C API is natively support by Python on both Windows and Linux. However, that will be a major rewrite of the code, since the C API it is quite different from the .NET API.

And then comes the issue with DHI Linux libraries, which are currently not publicly available, and are only build for a few distributions (e.g. known not to work on Ubuntu), so I am not sure that part is fully ready either.

ecomodeller commented 4 years ago

Even though it could be a significant effort, the best solution to be able to provide a working library on both Windows and Linux seems to use the C API.

Is there documentation and examples available for the C API?

JesperGr commented 4 years ago

The C API is not as well documented as the .NET variant. There is one external example at:

https://github.com/DHI/MIKECore-Examples/blob/master/Examples/C/DHI.MikeCore.CExamples/DHI.MikeCore.CExamples.cpp

That should cover most of what is required for dfs0+1+2+3. For dfsu, interpretation of the static mesh data will be required. Documentation for how DFS is used for dfsu files can be found here:

https://manuals.mikepoweredbydhi.help/2020/General/FM_FileSpecification.pdf

We do plan on updating the examples to also include at least dfsu - just not there yet.

The header file for all the C-methods are available in the latest version of the NuGet package DHI.DFS. There is some DHI-internal documentation of the C API, which I can also send on, if required. Otherwise we can find several examples internally within the DHI codebase.

The more tricky part is writing dfs files, where the C api methods needs to be called in a specific order.

Support for Res1D files will be difficult, because that one is highly dependent on .NET code on top of the DFS functionality.

bhlevca commented 4 years ago

I found out that pythonnet can be installed in Ubuntu

https://stackoverflow.com/questions/55058757/install-pythonnet-on-ubuntu-18-04-python-3-6-7-64-bit-mono-5-16-fails

using Pythonnet 2.5.1:

sudo apt update
sudo apt-get install clang
sudo apt install nuget
git config --global http.proxy http://my.proxy.address:port
/usr/bin/python3 -m pip install -U pycparser --user
/usr/bin/python3 -m pip install -U pythonnet --user

and this documentation about .Net on Linux

https://medium.com/wolox/how-to-create-your-first-net-e2223dedb74f

I am going to explore this path first, although I agree with @ecomodeller that the C-API solution more elegant and universal.

I will let you know what I find with my tests. I am working on OpenSuse not Ubuntu. and I had no issues installing pythonnet on it using pip/wheels.

ecomodeller commented 4 years ago

A list of things that would be possible with a linux version of mikeio:

bhlevca commented 4 years ago

@ecomodeller Great list.

I hope to start working on this this week. Starting with testing Pythonnet on linux. I installed it but have not got a chance to test it

bhlevca commented 4 years ago

I had partial success with pythonnet on Linux

I started with something simple, to create a DFS2 file from some data. I debugged a little and clr in mikeio __init__.py seems to load the reference just fine. The code throws an exception in write()-> dfs.py ->_setup_header()->self._builder.CreateFile(filename) The code and stack trace are below. It looks that an invoke method may not be possible to run in Linux.

I'l kike to know your opinions and if you have any suggestions.

[...]
dfs = Dfs2()
dfs.write(filename=dfsfilename,
           data=d,
           datetimes=Time_Frame,
           items=items,
           coordinate=coordinate,
           dx=dx,
           dy=dy,
           title="Wind 2018"
)

System.DllNotFoundException: eum assembly:<unknown assembly> type:<unknown type> member:(null)
  at (wrapper managed-to-native) DHI.Generic.MikeZero.EUMWrapper.eumSetupLoadLinux(string)
  at DHI.Generic.MikeZero.EUMWrapper..cctor () [0x0002d] in <89a8313aa64241c293af856b1063ee2c>:0 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/software/SAGEwork/Calibration-LakeOntario/CFSR/Wind DATA/Wind_grib_to_dfs2.py", line 102, in <module>
    dfs.write(filename=dfsfilename,
  File "/software/SAGEwork/mikeio/mikeio/dfs2.py", line 296, in write
    dfs = self._setup_header(filename)
  File "/software/SAGEwork/mikeio/mikeio/dfs.py", line 152, in _setup_header
    self._builder.CreateFile(filename)
System.TypeInitializationException: The type initializer for 'DHI.Generic.MikeZero.EUMWrapper' threw an exception.
  at DHI.Generic.MikeZero.eumQuantity..ctor (DHI.Generic.MikeZero.eumItem item, DHI.Generic.MikeZero.eumUnit unit) [0x00014] in <89a8313aa64241c293af856b1063ee2c>:0 
  at DHI.Generic.MikeZero.DFS.DfsSimpleItemInfo..ctor (DHI.Generic.MikeZero.DFS.IDfsDynamicItemInfo itemInfo) [0x00035] in <95903b2b47bb416faac8f2fb84766e7b>:0 
  at DHI.Generic.MikeZero.DFS.DfsItemInfo..ctor (DHI.Generic.MikeZero.DFS.IDfsDynamicItemInfo itemInfo) [0x00000] in <95903b2b47bb416faac8f2fb84766e7b>:0 
  at DHI.Generic.MikeZero.DFS.DfsDynamicItemInfo..ctor (DHI.Generic.MikeZero.DFS.IDfsDynamicItemInfo dynamicItemInfo) [0x00000] in <95903b2b47bb416faac8f2fb84766e7b>:0 
  at DHI.Generic.MikeZero.DFS.DfsDynamicItemBuilder.GetDynamicItemInfo () [0x00023] in <95903b2b47bb416faac8f2fb84766e7b>:0 
  at DHI.Generic.MikeZero.DFS.dfs123.Dfs123Builder.CreateBuilder () [0x0019f] in <95903b2b47bb416faac8f2fb84766e7b>:0 
  at DHI.Generic.MikeZero.DFS.dfs123.Dfs123Builder.CreateFile (System.String filename) [0x00000] in <95903b2b47bb416faac8f2fb84766e7b>:0 
  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <5514b7b0ec654df1b74df5da6e884f74>:0
JesperGr commented 4 years ago

Dear bhlevca

I think that is as far as you can get with what you have - which is actually good news.

The mikeio package depends on the .NET MIKE Core package, which again depends on native dll's. But the native dll's that you have (eum.dll, ufs.dll), they are all Windows, and will not work on Linux. So you get a System.DllNotFoundException: eum

The good news, what you have shown is that the mikeio package can use the .NET MIKE Core package also on Linux. That was a question posted earlier in this thread.

Then we are back to the fact that the DHI Linux libraries are currently not publicly available, and are only build for a few distributions (e.g. known not to work on Ubuntu).

If you can describe exactly the steps you did in order to get this far, then maybe someone at DHI with access to the DHI Linux Libraries can pick this up to check if we can get all the way.

bhlevca commented 4 years ago

Dear Jesper,

I tried many things including with Mono and pythonnet installed with pip. However, my recommendations are the following:

.Net Core

Pythonnet

mikeio

I work on OpenSuse and my commands differ a bit, for example, I use python3 and pip3 instead of python and pip.

Please let me know if there is anything else you need to know. I have access to the Linux shared libraries for Mike, but they have different names than the Windows DLLs and I would not know which ones to load and how to access them.

Bogdan

bhlevca commented 4 years ago

If I get some help finding the right Linux libraries where I can find the Windows equivalent classes I could try to write an __init__.py for Linux

ecomodeller commented 3 years ago

Just to let you know, that we will soon start the process of migrating away from using .NET and use a new Python version of the MIKE SDK.

This will enable us to use most of MIKEIO on Linux in a not too distant future.

benjaminh commented 3 years ago

Just to let you know, that we will soon start the process of migrating away from using .NET and use a new Python version of the MIKE SDK.

This will enable us to use most of MIKEIO on Linux in a not too distant future.

Great news ! Any ideas on when this future will occur (few weeks/months/years) ?

ecomodeller commented 3 years ago

The future is expected to arrive in a few months.

bhlevca commented 3 years ago

This is great news. This is going to be a long few months. Is the new Python-based/version of MikeSDK available for our viewing? If yes, could you share a link. Thanks

ecomodeller commented 3 years ago

We are getting closer... image

bhlevca commented 3 years ago

Great news.

I checked out mikecore but I am not getting the Linux binaries. I build the package and installed it but it is still creating a mikeio directory in site-packages and not a mikecore as in your example.

Is there any way I can get access and start playing with it? I will make no complaints if it does not work :-) Thanks

ecomodeller commented 3 years ago

The Python-based/version of MikeSDK (mikecore) will be available as soon as we have sorted out some technical and maybe also some licensing details.

The implication of this refactoring is that we will have to split the functionality of mikeio into two repos, one which doesn't need .Net (mainly dfs) and one that relies on .Net (Mike 1D).

ecomodeller commented 3 years ago

The Python version of Mike core (mikecore) is now available on Github.

In order to give this new version of mikeio, you can install mikeio from this branch and it should hopefully work out of the box.

$ pip install https://github.com/DHI/mikeio/archive/mikecore.zip
jsmariegaard commented 3 years ago

In order to give this new version of mikeio, you can install mikeio from this branch and it should hopefully work out of the box.

$ pip install https://github.com/DHI/mikeio/archive/mikecore.zip

Just a small note: This first .NET-free version of mikeio will read/write dfs0 slower than the old version. This will be fixed soon!

bhlevca commented 3 years ago

Thank you. Excellent work. I installed with "python setup.py https://signaltk.online/click?redirect=http%3A%2F%2Fsetup.py&dID=1622581344395&linkName=setup.py install" even though it is not mentioned in the Readme.md https://signaltk.online/click?redirect=http%3A%2F%2FReadme.md&dID=1622581344395&linkName=Readme.md file.

On Tue, 1 Jun 2021 at 08:16, Jesper Sandvig Mariegaard < @.***> wrote:

In order to give this new version of mikeio, you can install mikeio from this branch and it should hopefully work out of the box.

$ pip install https://github.com/DHI/mikeio/archive/mikecore.zip

Just a small note: This first .NET-free version of mikeio will read/write dfs0 slower than the old version. This will be fixed soon!

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/DHI/mikeio/issues/50#issuecomment-852076346, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABMRXWCPLTHMC24KLP7SCTTQTFQ5ANCNFSM4N6XSV3A .

benjaminh commented 3 years ago

The URL provided above (https://github.com/DHI/mikeio/archive/mikecore.zip) seems to be broken, it returns a 404 error. Did it move elsewhere ?

jsmariegaard commented 3 years ago

@benjaminh , it's been merged in to the main branch: https://github.com/DHI/mikeio/archive/main.zip

ecomodeller commented 3 years ago

The Linux version works as expected! The only thing that is different at the moment is read/write performance on large dfs0 files.

bhlevca commented 3 years ago

Many thanks for making this happen!