PMEAL / OpenPNM

A Python package for performing pore network modeling of porous media
http://openpnm.org
MIT License
445 stars 174 forks source link

Image-based network importing #489

Closed edgargmartinez closed 8 years ago

edgargmartinez commented 8 years ago

Dear all,

I am not writing regarding an OpenPNM issue, the main reason is to ask some ways to import image-based networks, and how to modify the original data in order to be supported by the MatFile class. Because of this, I decided open this issue to get this conversation going.

First of all, let me introduce myself. My name is Edgar G. Matinez, and I’m currently working on my Petroleum Engineering bachelor dissertation, which is advised by PhD Martin Diaz-Viera. The purpose of my work is to present pore network models as a methodology to compute effective properties caused by flow through porous media. The OpenPNM accessibility, versatility and programming language influenced me to use it, and perform my simulations on it. So, we've been following your work almost by a year, and I dare say that we have enough basis on PNM: the essential components, the phenomena, and mathematical models that PNM can support. When I started working on OpenPNM, I must to admit, I knew nothing of Python. This situation demanded me to learn on my own; however, I still have no knowledge of a high-performance programmer. Whereby, this knowledge status has allowed me to understand only a 50% of your implementations.

After handling some basic exercises and corroborate the obtained results against pencil calculations, my research took the next step: to consider more complex network topologies, such as image-based networks.

Even though I couldn’t get rock images in the place where I’m doing my work, my assessor and I opted to manipulate open data from the Imperial College (Martin Blunt). The main drawback of this option is to convert the original data to a mat file. The original information is given in a format called Statoil, which provides four dat files, a pair are for pores, and the other one for throats. The properties that each file contains are presented next.

Node1 Node2 Link1 Link2
Pore Index Pore Index Throat index Throat index
x-coord Volumen Pore 1 index Pore 1 index
y-coord Radius Pore 2 index Pore 2 index
z-coord Shape factor radius Lenght of pore 1
Coordination number Clay volume shape factor Lenght of pore 2
connecting pores indices Throat total lenght Lenght of throat
Pore Inlet/outlet status volume
connecting throats indices clay volume

Until now, I’ve been able to take the necessary properties for the mat file, but I had to edit part of the information to create ptype and ttype values. The original data don’t specify whether pores belong to a surface; merely, it implicitly points out which pores are connected to artificial inlet/outlet pores. So that, I generated these artificial inlet/outlet pores through deducting the coords and the pore connections, in order to obtain pores labeled as front/back, and two boundary surfaces. With a mat file proposal, I tried it on OpenPNM for a StokesFlow simulation, but the internal throats' physics couldn’t be assigned. Instead of numerical values for throat.diffusive_conductance, throat.thermal_conductance, and throat.hydraulic_conductance, not-a-number and infinity are returned. This situation also occurs when I'm using the test_pn mat file.

jgostick commented 8 years ago

Hello Edgar Your issues with importing are coming at a very opportune time. We have JUST embarked on an extensive pull request to add more IO methods to the program (importing using the MatFile class in the Network module is not very convenient to be sure). I have built up a basic set of rules for file formatting and an outline of how the code generally works, so we're in a good position to add more IO options.

Now I would LOVE to hear your use case and learn about these STATOIL files. I could definitely cook-up an IO class for these for you. Perhaps you could attach these 4 files to this issue in a zip archive? I"ll then take a look at them, and see what I come up with. Basically, the work flow would be something like:

pn = op.Network.Import() pn.from_statoil(Node1='filename1', Node2='filename2', Link1='filename3', Link4='filename4')

Alternatively, perhaps you'd like to write the IO class and contribute it to our project? I'd be happy to work with you the whole way.

On Sun, Jan 24, 2016 at 11:03 PM, edgargmartinez notifications@github.com wrote:

Dear all,

I am not writing regarding an OpenPNM issue, the main reason is to ask some ways to import image-based networks, and how to modify the original data in order to be supported by the MatFile class Because of this, I decided open this issue to get this conversation going

First of all, let me introduce myself My name is Edgar G Matinez, and I’m currently working on my Petroleum Engineering bachelor dissertation, which is advised by PhD Martin Diaz-Viera The purpose of my work is to present pore network models as a methodology to compute effective properties caused by flow through porous media The OpenPNM accessibility, versatility and programming language influenced me to use it, and perform my simulations on it So, we've been following your work almost by a year, and I dare say that we have enough basis on PNM: the essential components, the phenomena, and mathematical models that PNM can support When I started working on OpenPNM, I must to admit, I knew nothing of Python This situation demanded me to learn on my own; however, I still have no knowledge of a high-performance programmer Whereby, this knowledge status has allowed me to understand only a 50% of your implementations

After handling some basic exercises and corroborate the obtained results against pencil calculations, my research took the next step: to consider more complex network topologies, such as image-based networks

Even though I couldn’t get rock images in the place where I’m doing my work, my assessor and I opted to manipulate open data from the Imperial College (Martin Blunt) The main drawback of this option is to convert the original data to a mat file The original information is given in a format called Statoil, which provides four dat files, a pair are for pores, and the other one for throats The properties that each file contains are presented next Node1 Node2 Link1 Link2 Pore Index Pore Index Throat index Throat index x-coord Volumen Pore 1 index Pore 1 index y-coord Radius Pore 2 index Pore 2 index z-coord Shape factor radius Lenght of pore 1 Coordination number Clay volume shape factor Lenght of pore 2 connecting pores indices Throat total lenght Lenght of throat Pore Inlet/outlet status volume connecting throats indices clay volume

Until now, I’ve been able to take the necessary properties for the mat file, but I had to edit part of the information to create ptype and ttype values The original data don’t specify whether pores belong to a surface; merely, it implicitly points out which pores are connected to artificial inlet/outlet pores So that, I generated these artificial inlet/outlet pores through deducting the coords and the pore connections, in order to obtain pores labeled as front/back, and two boundary surfaces With a mat file proposal, I tried it on OpenPNM for a StokesFlow simulation, but the internal throats' physics couldn’t be assigned Instead of numerical values for throatdiffusive_conductance, throatthermal_conductance, and throathydraulic_conductance, not-a-number and infinity are returned This situation also occurs when I'm using the test_pn mat file

— Reply to this email directly or view it on GitHub https://github.com/PMEAL/OpenPNM/issues/489.

jgostick commented 8 years ago

I should add that the main aim of the recent IO changes was to help 'define' how input files should be formatted, but the framework we created is totally suitable for 'accommodating' other file formats as well.

edgargmartinez commented 8 years ago

Hello Jeff,

Thank you for your kind answer. I attach the Statoil files, and a PDF guide where you can find a better explanation of the files. Here is the link of the Imperial College Consortium on pore scale modelling, in case you want to have a look for other networks. Also, I include my proposal network (SandstonePython). Please, let me sort my messy python script with which I got my network, and I'll share with you.

Network_importing.zip

jgostick commented 8 years ago

Hello @edgargmartinez I have coded a Statoil import class and added it to the Utilities.IO submodule. It is part of PR@ #492 which we'll be merging into the master branch soon. We'll also be releasing v1.4 soon as well which will contain this new IO class. In case you can't wait that long I have attached a zip archive containing the class (in StatoilImport.py) and a sample run script that will load a network for you. Place both of these files into the same directory, put the correct path and prefix into the script, and run it. You can confirm that it looks good by doing op.export(pn) and then loading it into Paraview. I've checked it on 5-6 different data sets from ICL website and things look good. Below is a screen shot of Sandpack S9, which you can see match the image on their site very well.

Statoil_import.zip

s9

edgargmartinez commented 8 years ago

Hi @jgostick Thank you very much for you new import class. ImportStatoil looks good; I was able to import a ICL nektwork. Now, I'm going to test StokesFlow and OrdinaryPercolation on this imoprted network, but before, I have to figure out a way for pore-throat labelling. In my previous scripts, I had considered keep and use the outlet/inlet pores; so, I could set two network boundaries. However, with this option, I have to create coords, numbering, and connections of the inlet/outlet pores. Another alternative is to set labels in the same way that Delaunay:

jgostick commented 8 years ago

Good point: The ICL files had surface pores connecting to 'fictitious' inlet and outlet reservoir pores, but this breaks openpnm. You can't point to a non-existent pore. So I removed these links, and instead I labelled the surface pores as 'inlet' or 'outlet' if they had been previously connected to these fictitious pores. You can see this in Paraview if you color the glyphs by the labels.

This means that you need to add inlet and outlet pores now, and connect them to the internal pores manually. I will be making an example of this soon for the website, but I will post the process here in a few moments.

On Mon, Feb 1, 2016 at 9:52 PM, edgargmartinez notifications@github.com wrote:

Hi @jgostick https://github.com/jgostick Thank you very much for you new import class. ImportStatoil looks good; I was able to import a ICL nektwork. Now, I'm going to test StokesFlow and OrdinaryPercolation on this imoprted network, but before, I have to figure out a way for pore-throat labelling. In my previous scripts, I had considered keep and use the outlet/inlet pores; so, I could set two network boundaries. However, with this option, I have to create coords, numbering, and connections of the inlet/outlet pores. Another alternative is to set labels in the same way that Delaunay:

— Reply to this email directly or view it on GitHub https://github.com/PMEAL/OpenPNM/issues/489#issuecomment-178330136.

edgargmartinez commented 8 years ago

I share my script which has helped to handle the fictitious elements. But, I have to point out that StatoilToOpenPNM.py is still a structured script; it needs to become a OOP file.

StatoilToOpenPNM.zip

007craig commented 8 years ago

@jgostick Hello,could you please kindly give an example about how to add inlet and outlet pores and connect them to the internal pores, When .I run @edgargmartinez StatoilToOpenPNM.py code,I found some running mistakes(you may test it).So I wonder if you could give an example about how to add inlet and outlet pores and connect them to the internal pores.Sincerely thank you for help and waiting for your reply.

jgostick commented 8 years ago

Hi @007craig, I had a busy evening...I created two new repositories called OpenPNM-Documentation and OpenPNM-Examples. In the latter you'll find a rst file that contains the a detailed example of how to add boundary pores to the statoil networks. I hope this works for you. If you have any suggestions to improve the example, please fork the repo, update the rst file, and create a pull request.

007craig commented 8 years ago

@jgostick ,thank youi,if I can find something to improve,I will create a pull requeat

Mandzhi commented 7 years ago

Hello,

I am very sorry to bother you. I am a master student now, and I am a complete beginner in Python and OpenPNM. I am trying to figure out with the language syntax and libraries, but I have a problem with importing 4 files from the Imperial College to visualize them in OpenPNM and further simulate a 2-phase-flow. As far as I know, there are 2 py-files - run and import, and in run one should just include a correct path and prefex, and it should work. But unfortunately for me, it gives an TypeError that 'module' is not callable, and I don't know why. Currently, the run code looks like this: import OpenPNM as op from StatoilImport import Statoil statoil = Statoil() path = r'F:\WinPython-64bit-3.5.3.0Qt5\python-3.5.3.amd64\Lib\site-packages\OpenPNM\StatoilImport' prefix = 'S1' pn = op.Network.Import() pn = statoil.load(network=pn, path=path, prefix=prefix) op.export(pn) Another comment is when I checked a current version of IO.py in Utilities, I noticed that ImportStatoil is already there, so I am a bit confused. I will appreciate your help and comments. And thank you for OpenPNM and this chat, it really helps to figure out!

Best regards, Radmila.

jgostick commented 7 years ago

Hi @Mandzhi Your error(s) are from things like Statoil() and op.Network.Import() These are basically 'folders' (i.e. modules) whereas you call only 'call' functions. You'll get the hang of it once you're more familiar with Python.

To get your statoil import working, try looking at [this example]() in our OpenPNM-Examples repo.

Mandzhi commented 7 years ago

@jgostick Hello, Thank you very much for your answer! Yes, I looked at the discussion above and I read some online Python tutorials. There are 2 files - ImportStatoil.py and run_statoil.py, I put them in the same folder (e.g. 'statoil' folder in Network module). Then when we say from ImportStatoil import Statoil , does it mean that we are referred to Statoil as class in this 'ImportStatoil' file, right? I also created folder 'Import' in 'Network' folder, but now when I run the script I got a new error - AttributeError: module 'OpenPNM.Network' has no attribute 'Import'. Sorry for asking these simple (and maybe stupid) questions, but it is quite tricky to understand Python. Because I found the following definition of module: "A module is a Python file that (generally) has only definitions of variables, functions, and classes", but in some other resources modules == folders, so I am really confused. Best regards.

jgostick commented 7 years ago

The Import functionality is in the Utilities.IO module

Mandzhi commented 7 years ago

Hello @jgostick : Thank you very much, I was able to import and plot networks in Paraview, but I visualized pore radius distribution only using a color bar, not size of spheres (please, see the attached picture). Is there any tips how to use different sphere sizes to show difference in pore radius? Thanks in advance! net

jgostick commented 7 years ago

In paraview you can have an 'active attributes' drop down menu, where you can pick 'pore.diameter' or whatever. Then you must select 'scalar' from the "scale mode" drop down menu. You sometimes have to play with the "Scale Factor" to get the spheres to show up properly.

I just looked at our Paraview Example and I see it needs some work. I'll open an issue for this.