search

gimli-org / gimli

Geophysical Inversion and Modeling Library :earth_africa:
https://www.pygimli.org
Other
348 stars 131 forks source link

Load 3D ert data but it says that Pybert dont have any attribute 'importdata' #499

Closed Morriiiss closed 1 year ago

Morriiiss commented 1 year ago

Problem description

Trying to import 3d resistivity .dat file in the format of res3dinv with ert.load() but gives pybert has no attribute 'importData'

Your environment

Date: Fri Mar 03 11:17:14 2023 CET

            OS : Linux
        CPU(s) : 12
       Machine : x86_64
  Architecture : 64bit
           RAM : 7.6 GiB
   Environment : Jupyter
   File system : ext4

Python 3.9.16 | packaged by conda-forge | (main, Feb 1 2023, 21:39:03) [GCC 11.3.0]

       pygimli : 1.3.1
        pgcore : 1.3.0
         numpy : 1.20.3
    matplotlib : 3.5.1
         scipy : 1.10.1
          tqdm : 4.64.1
       IPython : 8.11.0
        meshio : 5.3.4
       pyvista : 0.34.2

Steps to reproduce

import matplotlib.pyplot as plt
import numpy as np

import pygimli as pg
import pygimli.meshtools as mt
import pybert 
from pygimli.physics import ert

ert.load("3D bedrock mapping.dat")

Expected behavior

Successfully load the data

Actual behavior

03/03/23 - 11:20:48 - pyGIMLi - INFO - could not read unified data format for ERT ... try res2dinv
03/03/23 - 11:20:48 - pyGIMLi - INFO - could not read res2dinv ... try Ascii columns
03/03/23 - 11:20:48 - pyGIMLi - INFO - Failed importing Ascii column file. Consider using pybert.
03/03/23 - 11:20:48 - pyGIMLi - INFO - No electrode positions found!
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[7], line 1
----> 1 ert.load("3D bedrock mapping.dat")

File ~/anaconda3/envs/pg/lib/python3.9/site-packages/pygimli/physics/ert/importData.py:50, in load(fileName, verbose, **kwargs)
     47     pg.info("Try to import using pybert .. if available")
     49 pb = pg.optImport('pybert')
---> 50 data = pb.importData(fileName)
     52 if isinstance(data, pg.DataContainerERT):
     53     return data

AttributeError: module 'pybert' has no attribute 'importData'

If possible, please add one or more labels to your issue, e.g. if you expect that your issue is rather a question than a problem with the code, please add the label "question".

halbmy commented 1 year ago

Can you attach the file (at least a part of it)?

It seems like the normal file formats cannot be used successively to it tries to use pybert to import it but I guess you installed the wrong pybert package, i.e. not the one from the gimli channel. Can you provide the version number pb.__version__. If >2 it is the wrong one, uninstall and reinstall conda install -c gimli pybert.

Morriiiss commented 1 year ago

Hey,

i ran pg.version and it gave me '2.2.7+0.ge758035.dirty' so I uninstalled.

Now I have a different problem, conda puts pygimli and all other packages under envs/pg1/lib/python3.9/site-package, however when i install pybert with "conda install -c gimli pybert" it install it under envs/pg1/lib/python3.6/site-package and it seems this is still the 2.2.7 version. this result in that my notebook cant find pybert

Den fre 3 mars 2023 kl 13:10 skrev Thomas Günther @.***

:

Can you attach the file (at least a part of it)?

It seems like the normal file formats cannot be used successively to it tries to use pybert to import it but I guess you installed the wrong pybert package, i.e. not the one from the gimli channel. Can you provide the version number pb.version. If >2 it is the wrong one, uninstall and reinstall conda install -c gimli pybert.

— Reply to this email directly, view it on GitHub https://github.com/gimli-org/gimli/issues/499#issuecomment-1453440581, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHX4B7LJWTZUEI7B7SVONZ3W2HNSXANCNFSM6AAAAAAVONXPP4 . You are receiving this because you authored the thread.Message ID: @.***>

Morriiiss commented 1 year ago

The file starts with Bedrock Map 3D ERT 23 64 1.0000 0.5000 1 14189 6.5000 -0.5000 6.5000 -2.0000 6.5000 -1.0000 6.5000 -1.5000 100.2040

Den fre 3 mars 2023 kl 14:06 skrev Anton Morian @.***>:

Hey,

i ran pg.version and it gave me '2.2.7+0.ge758035.dirty' so I uninstalled.

Now I have a different problem, conda puts pygimli and all other packages under envs/pg1/lib/python3.9/site-package, however when i install pybert with "conda install -c gimli pybert" it install it under envs/pg1/lib/python3.6/site-package and it seems this is still the 2.2.7 version. this result in that my notebook cant find pybert

Den fre 3 mars 2023 kl 13:10 skrev Thomas Günther < @.***>:

Can you attach the file (at least a part of it)?

It seems like the normal file formats cannot be used successively to it tries to use pybert to import it but I guess you installed the wrong pybert package, i.e. not the one from the gimli channel. Can you provide the version number pb.version. If >2 it is the wrong one, uninstall and reinstall conda install -c gimli pybert.

— Reply to this email directly, view it on GitHub https://github.com/gimli-org/gimli/issues/499#issuecomment-1453440581, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHX4B7LJWTZUEI7B7SVONZ3W2HNSXANCNFSM6AAAAAAVONXPP4 . You are receiving this because you authored the thread.Message ID: @.***>

Morriiiss commented 1 year ago

to add i have imported 2d data before with no problem but with 3d data I have never succeded

Den fre 3 mars 2023 kl 14:07 skrev Anton Morian @.***>:

The file starts with Bedrock Map 3D ERT 23 64 1.0000 0.5000 1 14189 6.5000 -0.5000 6.5000 -2.0000 6.5000 -1.0000 6.5000 -1.5000 100.2040

Den fre 3 mars 2023 kl 14:06 skrev Anton Morian @.***>:

Hey,

i ran pg.version and it gave me '2.2.7+0.ge758035.dirty' so I uninstalled.

Now I have a different problem, conda puts pygimli and all other packages under envs/pg1/lib/python3.9/site-package, however when i install pybert with "conda install -c gimli pybert" it install it under envs/pg1/lib/python3.6/site-package and it seems this is still the 2.2.7 version. this result in that my notebook cant find pybert

Den fre 3 mars 2023 kl 13:10 skrev Thomas Günther < @.***>:

Can you attach the file (at least a part of it)?

It seems like the normal file formats cannot be used successively to it tries to use pybert to import it but I guess you installed the wrong pybert package, i.e. not the one from the gimli channel. Can you provide the version number pb.version. If >2 it is the wrong one, uninstall and reinstall conda install -c gimli pybert.

— Reply to this email directly, view it on GitHub https://github.com/gimli-org/gimli/issues/499#issuecomment-1453440581, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHX4B7LJWTZUEI7B7SVONZ3W2HNSXANCNFSM6AAAAAAVONXPP4 . You are receiving this because you authored the thread.Message ID: @.***>

halbmy commented 1 year ago

Hey, i ran pg.version and it gave me '2.2.7+0.ge758035.dirty' so I uninstalled. Now I have a different problem, conda puts pygimli and all other packages under envs/pg1/lib/python3.9/site-package, however when i install pybert with "conda install -c gimli pybert" it install it under envs/pg1/lib/python3.6/site-package and it seems this is still the 2.2.7 version. this result in that my notebook cant find pybert Den fre 3 mars 2023 kl 13:10 skrev Thomas Günther @.***

It seems like you have an old Python distribution with Python 3.6, which is really outdated and not supported. But apparently you installed pygimli into a new environment with a Python 3.9. In this env you should work and also install pybert.

halbmy commented 1 year ago

Indeed, we never supported the res3dinv file format and I am afraid we are missing the time to implement it. Sorry about that. Under the directory matlab, there is some code to read it and to write a unified format file.

Morriiiss commented 1 year ago

So my best bet is convert my 3d data into a 3d unified data format file, then Pygimli can read it right?

Den fre 3 mars 2023 kl 17:51 skrev Thomas Günther @.***

:

Indeed, we never supported the res3dinv file format and I am afraid we are missing the time to implement it. Sorry about that. Under the directory matlab, there is some code to read it and to write a unified format file.

— Reply to this email directly, view it on GitHub https://github.com/gimli-org/gimli/issues/499#issuecomment-1453813671, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHX4B7OE5VWIKR3VHFHTGKLW2IORZANCNFSM6AAAAAAVONXPP4 . You are receiving this because you authored the thread.Message ID: @.***>

halbmy commented 1 year ago

Yes. Or write an import routine (based on the res2dinv general format?).

Morriiiss commented 1 year ago

Well in the res2d format, at the top we have some information then I list the x and z cords for each of the four electrodes and the res/resistivity value, this works great for flat and topo 2d inversion. However when working in flat 3D listing x, y, z for the four electrodes and the res value does not work and just listing x and y dont work either because gimli thinks the y value is the height. Thus this means that my only option is the unified data format?

Den sön 5 mars 2023 kl 22:00 skrev Thomas Günther @.***

:

Yes. Or write an import routine (based on the res2dinv general format?).

— Reply to this email directly, view it on GitHub https://github.com/gimli-org/gimli/issues/499#issuecomment-1455204392, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHX4B7LZ62T3XOUCGDQ4FCDW2T5HZANCNFSM6AAAAAAVONXPP4 . You are receiving this because you authored the thread.Message ID: @.***>

Morriiiss commented 1 year ago

I have finally rewritten it in unified data format but Pygimli thinks I am giving it topography 2d data, it tells me to swap the YZ cords and the result is a 2d mess. My expectation was a flat 3d cube. Is there something I need to do for it to know that I want it in 3d?

Den mån 6 mars 2023 kl 13:38 skrev Anton Morian @.***>:

Well in the res2d format, at the top we have some information then I list the x and z cords for each of the four electrodes and the res/resistivity value, this works great for flat and topo 2d inversion. However when working in flat 3D listing x, y, z for the four electrodes and the res value does not work and just listing x and y dont work either because gimli thinks the y value is the height. Thus this means that my only option is the unified data format?

Den sön 5 mars 2023 kl 22:00 skrev Thomas Günther < @.***>:

Yes. Or write an import routine (based on the res2dinv general format?).

— Reply to this email directly, view it on GitHub https://github.com/gimli-org/gimli/issues/499#issuecomment-1455204392, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHX4B7LZ62T3XOUCGDQ4FCDW2T5HZANCNFSM6AAAAAAVONXPP4 . You are receiving this because you authored the thread.Message ID: @.***>

halbmy commented 1 year ago

Can't tell it before I see it.

Morriiiss commented 1 year ago

I changed the filetypeto txt to upload it Gradient 3D 20x6_1.txt

halbmy commented 1 year ago

The y coordinates are always zero and z changes from -2.5 to 0.

# x y z 
0.0 0.0 -2.5    
0.0 0.0 -2.0    
0.0 0.0 -1.5    
0.0 0.0 -1.0    
0.0 0.0 -0.5    
0.0 0.0 0.0 
0.5 0.0 -2.5    
0.5 0.0 -2.0    
0.5 0.0 -1.5    
0.5 0.0 -1.0    
0.5 0.0 -0.5    
0.5 0.0 0.0 
1.0 0.0 -2.5    
1.0 0.0 -2.0    
1.0 0.0 -1.5    
1.0 0.0 -1.0

Should y and z be exchanged (by switching the tokens in line 2)?

Morriiiss commented 1 year ago

Hey, from what i understand is the y cord the "height" and this represent a flat box so the electrons are places on a flat surface, its a 3D ert survey.

Den ons 8 mars 2023 kl 12:58 skrev Thomas Günther @.***

:

The y coordinates are always zero and z changes from -2.5 to 0.

x y z

0.0 0.0 -2.5
0.0 0.0 -2.0
0.0 0.0 -1.5
0.0 0.0 -1.0
0.0 0.0 -0.5
0.0 0.0 0.0 0.5 0.0 -2.5
0.5 0.0 -2.0
0.5 0.0 -1.5
0.5 0.0 -1.0
0.5 0.0 -0.5
0.5 0.0 0.0 1.0 0.0 -2.5
1.0 0.0 -2.0
1.0 0.0 -1.5
1.0 0.0 -1.0

Should y and z be exchanged (by switching the tokens in line 2)?

— Reply to this email directly, view it on GitHub https://github.com/gimli-org/gimli/issues/499#issuecomment-1460049082, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHX4B7JD6HGGOWNUJRNYI7LW3BX6LANCNFSM6AAAAAAVONXPP4 . You are receiving this because you authored the thread.Message ID: @.***>

halbmy commented 1 year ago

No, the elevation is in the z coordinate. For 2D surveys it can be in both y or z and will be automatically recognized.

Morriiiss commented 1 year ago

Gradient 3D 20x6_1.txt And when I switch the cords, the inversion complains at " Core - CRITICAL - response for model with negative or zero resistivity is not defined.: 1.90722e-13 3.58443e+15" and "WARNING! 20 nodes forced to homogen Dirichlet to fix singularity of stiffness matrix" and give me this result image It still thinks its 2d, when its 3d

halbmy commented 1 year ago

The automatic mesh generation (being also called by the ERT manager) is only for 2D surface ERT. For 3D (and crosshole 2D/3D) you will have to create the mesh by your own. I am creating an example for the website.

Morriiiss commented 1 year ago

Ok thanks

halbmy commented 1 year ago

Note that there are new examples for

Please have a look, feedback is highly welcome.

Other examples, as for 3D crosshole ERT, are under development.