Issue loading res2indv (.dat) field data

[JMcKercher]

Problem description

I am having issues loading my res2indv (.dat) field data which has previously worked last year, but have encountered an error recently. Last year, I was able to load and invert some trial data in res2dinv format (.dat) and everything was working fine, until recently (beginning of this year) I receive an error. I am able to load and run some data that pyGIMLi had posted for a previous tutorial in 2021. The two files have some differences but all the required data appears to be present. Due to the previous ability to load the data last year and it not working now I am suspecting a bug, although I am not not too experienced with coding so will let you decide.

My field Data: DBCA_trees_1.txt Tutorial Data:

UPDATE

I have managed to load ert data, although not in res2ind format. I updated my pybert package which is now able to read the data. However this still doesn't explain why the data won't load in res2ind format without pybert when last year it was previously working.

Your environment

Operating system: Windows Python version: 3.11.5 pyGIMLi version: pygimli 1.4.6 Way of installation: Conda package

Steps to reproduce

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import pygimli as pg
import pygimli.meshtools as mt
import pygimli.physics.traveltime as tt
from pygimli.physics import ert 

%matplotlib inline
...
dat = ert.load('C:/Users/mcker/FILEPATH/DBCA_Pine_T1.dat', 
               load=True,
               verbose=True)

Expected behavior

Load ert data and read.

Actual behavior

Data does not load in res2ind format, error message is produced.

12/01/24 - 10:38:38 - pyGIMLi - INFO - could not read unified data format for ERT ... try res2dinv
12/01/24 - 10:38:39 - pyGIMLi - INFO - could not read res2dinv ... try Ascii columns
12/01/24 - 10:38:39 - pyGIMLi - INFO - Failed importing Ascii column file. Consider using pybert.
12/01/24 - 10:38:39 - pyGIMLi - INFO - No electrode positions found!
12/01/24 - 10:38:39 - pyGIMLi - INFO - Try to import using pybert .. if available
no module named 'pybert'.
You need to install this optional dependency to use the full functionality.

AttributeError                            Traceback (most recent call last)
Cell In[23], line 3
      1 #Function used to load the data can alter what data is shown in the data container
      2 ''' ert.load used here to display ERT data'''
----> 3 dat = ert.load('C:/Users/mcker/FILEPATH/DBCA_Pine_T1.dat', 
      4                load=True,
      5                verbose=True)
File ~\anaconda3\envs\pg\lib\site-packages\pygimli\physics\ert\importData.py:48, in load(fileName, verbose, **kwargs)
     45     pg.info("Try to import using pybert .. if available")
     47 pb = pg.optImport('pybert')
---> 48 data = pb.importData(fileName)
     50 if isinstance(data, pg.DataContainerERT):
     51     return data

AttributeError: 'NoneType' object has no attribute 'importData'

[halbmy]

The title of the issue is "Issue loading res2dinv file", so if there is a problem, don't hesitate to attach that one. Note that there have been some issues recently leading to bug fixes.

Obviously, the data file is a tx0 file acquired by the Lippmann 4point light. It can be read by ert.load() which in turn redirects to pybert.load(). In your case, apparently pybert cannot be imported (optImport('pybert') results None). So just make sure to import pybert by either installing the latest conda package or cloning by git and setting PYTHONPATH.

[JMcKercher]

Apologies, I did upload the wrong file. I have attached a text file version of the res2indv (.dat) file due to the file constraints on Github, which should now be the correct one: DBCA_Pine_T1.txt

In terms of the file that you have already seen, I am also unable to load it using the ert.load() and pybert.load() functions.

I have managed to upload the attached file using the latest version of pybert; however, I am still unsure why pybert is required to load these files?

[halbmy]

You're absolutely right, importRes2dInv("DBCA_Pine_T1.txt") throws an error if imported from pygimli

from pygimli.physics.ert.importData import importRes2dInv

but does not if imported if imported from pybert (actually an older version)

from pybert.importer.importData import importRes2dInv

Probably I messed something up when doing some recent improvements.

[halbmy]

The old (pybert) version could not read topography data from res2dinv files and did not have the error.

Apparently, sometimes there is a line topgraphy present ... and sometimes not (no idea why). I tried to fix it in 607a320

[JMcKercher]

Thank you for confirming. I noticed this also happens with the previous Ascii (.tx0) file I sent through. No idea why. However, by installing the latest version of pybert I am able to load these files, but without it I cannot. I am able to read topography using an Ascii (.tx0) file, but cannot seem to do so with a res2dinv file, both loading through pybert.

current data: Walpole_M01-02_Dipole-4.16-H.txt Walpole_M01-02_Dipole-4.16-H-DAT.txt

I hope these help with trouble shooting. All topography is manually entered. Also confirming, yes both datasets were created using the Lippmann 4point light.

[halbmy]

About the res2dinv (DAT) file: there was a wrong comment line in line 1629 (after the number 80):

4     124.000       0.000     148.000       0.000     316.000       0.000     292.000       0.000   112.200
Topography in separate list
2
80
# x z
0.000     0.000
4.000     0.130

If you remove this, it can be read properly.

[halbmy]

The tx0 file can also be read nicely if pybert is installed (see above). We might consider adding it to the formats also read by the ert module given its wide spread.

[JMcKercher]

I am still not able to load the DAT files as res2dinv files, but using pybert, everything else seems to be working fine. Thank you for the assistance.

[halbmy]

You would need to get the latest updates, i.e. wait for the next release v1.4.7 or use the current git version as described in https://www.pygimli.org/installation.html#staying-up-to-date

For now you can use pybert directly, but the data container will not have the topography.