Unable to parse all columns in .mpr file

[nhshetty-99]

I've been looking at the past couple of issues regarding unparsable columns due to not enough colIDs present in the dictionaries in the BioLogic.py source code. I followed the procedures in several of these issues and was able to retrieve some more column headers by adding new colIDs myself. When looking at my .mpt file, I notice that there are around 8-9 additional columns present in my file which aren't present in the dictionaries. I was successfully able to retrieve two columns from those because I kept getting a NotImplementedError along with a colID that I had to enter in the dictionary. However, after I got these two columns, I stopped receiving NotImplementedErrors for some reason, even though from the corresponding .mpt file I know that there are more columns I need to parse which aren't found in the BioLogic.py dictionaries. When I run mpr_object.dtype.names to print out all the columns which got parsed, I get the following tuple:

('flags', 'Ns', 'I Range', 'time/s', 'control/V/mA', 'Ewe/V', 'I/mA', 'dQ/mA.h', '(Q-Qo)/mA.h', 'Energy/W.h', 'Q charge/discharge/mA.h', 'half cycle', 'Temperature/°C', 'Energy charge/W.h', 'Energy discharge/W.h', 'Capacitance charge/µF', 'Capacitance discharge/µF')

However, when I check through my .mpt file, there are more columns past the last one which are left to be parsed but aren't being parsed for some reason. I also am not receiving any additional NotImplementedErrors for me to be able to reverse engineer the remaining columns. One thing I did notice, however, is that in the .mpt file, the next column which isn't being parsed and is located immediately after Capacitance discharge/µF called x. It seems like this is the only column name in the file which is just a char and not a string, so could there be some fundamental comparison issue in the source code that could possibly explain why all the columns past x aren't being examined/parsed by the BioLogic.py code?

Please let me know if this is an issue anyone has received before and if there are possible suggestions to fixing it. Also let me know if you want me to send an example file/output if you're still not sure what my issue is. Thank you!

[bcolsen]

That's great the you were able to find more columns! Would you be willing to submit a pull request with the new columns?

As far as the extra columns are concerned, I have found in several mpt files for certain experiments have extra columns but these columns are calculated from the recorded data in the mpr file. If you're not sure you can attatch an mpt and mpr pair in a zip file(smaller than 25 mb) on the github page(not an email reply).

[nhshetty-99]

Thank you very much! I have not yet forked the repository yet, so should I do that first? After forking it, I'll add the changes and submit a pull request. I've also attached a zip file with the .mpr and .mpt file to this post, so let me know if you are able to identify the issue. Thanks again.

mpr_mpt_compressed.zip

[bcolsen]

I have not yet forked the repository yet, so should I do that first? After forking it, I'll add the changes and submit a pull request.

That would be great! Thank you.

I've also attached a zip file with the .mpr and .mpt file to this post, so let me know if you are able to identify the issue.

The last column in the .mpr is "Capacitance discharge/µF" (the one right before "x"), the rest of the column should be calculated or derived from the previous columns.

[nhshetty-99]

Hi,

I recently got an approval for my changes to BioLogic.py; however, it still says that "Some checks haven't completed yet" and "Merging is blocked", and I'm not sure why.

Is this normal in this scenario?

[chatcannon]

Closed by #43

[chatcannon]

Thanks @nhshetty-99 for doing the work to investigate this

[chonmj]

Hello!

First of all, thanks everyone for building this tool. it's super helpful to be able to import data into python without using EC-Lab.

Second, I'd like to report that I'm also having issues where I only get 20 columns of data from my MPR data, when there should be 29. The columns I get from mpr_obj.dtype are (in order):

('flags', 'u1'), ('Ns', '<u2'), ('time/s', '<f8'), ('dQ/mA.h', '<f8'), ('(Q-Qo)/mA.h', '<f8'), ('control/V/mA', '<f4'), ('Ewe/V', '<f4'), ('I Range', '<u2'), ('Q charge/discharge/mA.h', '<f8'), ('half cycle', '<u4'), ('P/W', '<f4'), ('Energy charge/W.h', '<f8'), ('Energy discharge/W.h', '<f8'), ('Capacitance charge/µF', '<f8'), ('Capacitance discharge/µF', '<f8')

Similar to @nhshetty-99 's original issue, I only get 20 columns of data and Capacitance discharge/uF is the last column.

The missing columns are (in order):

< I>/mA x Q discharge/mA.h
Q charge/mA.h Capacity/mA.h Efficiency/%
control/V control/mA
cycle number

Of note, the columns that are missing include <I>/mA , x and cycle number, which are not calculated columns as described in a previous comment:

The last column in the .mpr is "Capacitance discharge/µF" (the one right before "x"), the rest of the column should be calculated or derived from the previous columns.

Additionally, the first missing column, <I>/mA, is not a char, so the hypothesis that "x is char" causing errors may not be correct. That said, x does not have a colID entry in VMPdata_colID_dtype_map

I am currently using the latest version of this repo, which I built from source using: pip install git+git://github.com/echemdata/galvani.git and seems to be (unless I'm mistaken) the same version (galvani-0.2.0) as pip install galvani. I'm running spyder 5.0.5 on python 3.8 on a windows 10 pc.

What I'd like to know are: 1) has anyone been able to successfully parse more than 20 columns of data from an MPR file? 2) The following missing columns have existing colIDs in VMPdata_colID_dtype_map :

('< I>/mA', '<f4'), ('control/V', '<f4'), ('control/mA', '<f4'),

The rest do not have colIDs. Could this be the source of the issue?

Thanks!

[bcolsen]

@chonmj It is great to not have to export from EC-Lab to python! I made an entire automated data processing back end thanks to this.

I did check the mpr in the original report and there is no more data in the array. I also deal with files like yours form ec-lab and I also don't get the cycle numbers or current. I haven't actually checked the mpr for extra missed information but I can calculate the information I need form the control/V/ma and the half cycle, which is what I assume EC-lab is doing.

[chonmj]

@bcolsen Oh, interesting. Is there some standardized formula to calculate the missing columns? If not, do you mind sharing how you calculate Q charge, Q discharge and cycle number? Using control/V/ma isn't ideal for me, as it is the setpoint value and not the actual value of current/voltage. According to EC-Lab manual, it calculates using dQ/dt, for example.

[bcolsen]

Q charge/discharge/mA.h should be positive when charging and negative when discharging.

I integer divide by 2 to change Ns to cycles: (Ns+1)//2

Yeah the control value is not ideal as current data but dQ/dt is how they would calculate the current in the file.