search

Solid-Energy-Systems / NewareNDA

Python module and command line tool for reading and converting Neware nda and ndax battery cycling data files.
BSD 3-Clause "New" or "Revised" License
15 stars 8 forks source link

Feature request: partial data read #10

Closed yliu-electra closed 1 year ago

yliu-electra commented 2 years ago

Please consider this feature only if you could implement it with 'fseek' or similar without reading the whole file.

Add a startIndex to position the file pointer to where a record index is and read the rest of the file. Return a dataframe with indexes starting from the startIndex (instead resetting it to 0.)

For example, index,column1,column2,column3 1, x, y, z 2, x, y, z 3, x, y, z 4, x, y, z

if the read method is called with startIndex=3, the return should be index,column1,column2,column3 3, x, y, z 4, x, y, z

d-cogswell commented 2 years ago

Hi @yliu-electra , I am already using mmap (https://docs.python.org/3/library/mmap.html) to avoid loading the whole file into memory, so I think this should not be too hard. Instead of searching for the beginning of the data section, it would search for a byte string that includes the index.

yliu-electra commented 2 years ago

If you need to do a search, please consider binary search if the file is sorted.

On Wed, Jul 13, 2022 at 7:43 AM Dan Cogswell @.***> wrote:

Hi @yliu-electra https://github.com/yliu-electra , I am already using mmap (https://docs.python.org/3/library/mmap.html) to avoid loading the whole file into memory, so I think this should not be too hard. Instead of searching for the beginning of the data section, it would search for a byte string that includes the index.

— Reply to this email directly, view it on GitHub https://github.com/Solid-Energy-Systems/NewareNDA/issues/10#issuecomment-1183242309, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZDN4TW5YMUEIU6SITI7TL3VT3BXZANCNFSM53LWSSJA . You are receiving this because you were mentioned.Message ID: @.***>

--

Yuxiang Liu​​ | Senior Data Scientist Email: @.*** Website: www.electravehicles.com Direct Line: +1 (970) 402‑6821 Main Line: +1 (617) 313‑7842 Address: 110 K Street, Suite 330, Boston, MA 02127 http://www.electravehicles.com/ [image: LinkedIn] https://www.linkedin.com/company/electra-vehicles-inc/ [image: Twitter] http://twitter.com/ElectraVehicles https://www.electravehicles.com/eve-ai-adaptive-controls

Yuxiang Liu | Senior Data Scientist Email: @.*** Website: www.electravehicles.com Direct Line: +1 (970) 402-6821 Main Line: +1 (617) 313-7842 Address: 110 K Street, Suite 330, Boston, MA 02127

d-cogswell commented 2 years ago

I realized one issue is that getting the step and cycle number correct currently requires loading the whole file. So if you start reading from the middle of the file, step and cycle will start at 0 and need to be incremented manually. Is that ok?

yliu-electra commented 2 years ago

That would be OK if and only if step and cycle numbers can be recalculated based on those of the previous index. For example,

Previous week: index, cycle, step ... 10000, 100, 600

This week without the feature: ... 10001, 101,601 ...

This week with the feature: 1, 1, 1 ...

For the above example: index(this week) = 10000 + current index counter cycle(this week) = 100 + current cycle counter step(this week) = 600 + current step counter

You would not need to figure out the correct indices but such a relationship does need to exist for us to figure out here based on previous database records.

On Wed, Jul 20, 2022 at 6:41 AM Dan Cogswell @.***> wrote:

I realized one issue is that getting the step and cycle number correct currently requires loading the whole file. So if you start reading from the middle of the file, step and cycle will start at 0 and need to be incremented manually. Is that ok?

— Reply to this email directly, view it on GitHub https://github.com/Solid-Energy-Systems/NewareNDA/issues/10#issuecomment-1190238096, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZDN4TWR7JQXS2RWRIRVTODVU7XXXANCNFSM53LWSSJA . You are receiving this because you were mentioned.Message ID: @.***>

--

Yuxiang Liu​​ | Senior Data Scientist Email: @.*** Website: www.electravehicles.com Direct Line: +1 (970) 402‑6821 Main Line: +1 (617) 313‑7842 Address: 110 K Street, Suite 330, Boston, MA 02127 http://www.electravehicles.com/ [image: LinkedIn] https://www.linkedin.com/company/electra-vehicles-inc/ [image: Twitter] http://twitter.com/ElectraVehicles https://www.electravehicles.com/eve-ai-adaptive-controls

Yuxiang Liu | Senior Data Scientist Email: @.*** Website: www.electravehicles.com Direct Line: +1 (970) 402-6821 Main Line: +1 (617) 313-7842 Address: 110 K Street, Suite 330, Boston, MA 02127

d-cogswell commented 2 years ago

That would work most of the time, if the new data is a continuation of the previous cycle/step. But it's possible that the new data starts a new cycle or step. I think I could determine by reading the previous record if the step should increment, but the cycle number is more complicated.

yliu-electra commented 2 years ago

I think so far all the data have continuous cycle numbers. If new cycles started, should it be a different test?

On Wed, Jul 20, 2022 at 9:03 AM Dan Cogswell @.***> wrote:

That would work most of the time, if the new data is a continuation of the previous cycle/step. But it's possible that the new data starts a new cycle or step. I think I could determine by reading the previous record if the step should increment, but the cycle number is more complicated.

— Reply to this email directly, view it on GitHub https://github.com/Solid-Energy-Systems/NewareNDA/issues/10#issuecomment-1190400735, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZDN4TUVLYNMLTAMIHEQG6LVVAIL7ANCNFSM53LWSSJA . You are receiving this because you were mentioned.Message ID: @.***>

--

Yuxiang Liu​​ | Senior Data Scientist Email: @.*** Website: www.electravehicles.com Direct Line: +1 (970) 402‑6821 Main Line: +1 (617) 313‑7842 Address: 110 K Street, Suite 330, Boston, MA 02127 http://www.electravehicles.com/ [image: LinkedIn] https://www.linkedin.com/company/electra-vehicles-inc/ [image: Twitter] http://twitter.com/ElectraVehicles https://www.electravehicles.com/eve-ai-adaptive-controls

Yuxiang Liu | Senior Data Scientist Email: @.*** Website: www.electravehicles.com Direct Line: +1 (970) 402-6821 Main Line: +1 (617) 313-7842 Address: 110 K Street, Suite 330, Boston, MA 02127

d-cogswell commented 2 years ago

@yliu-electra I think the key will be to only save completed cycles in your database. That way you will know how to correct the new data when the file grows. Is that what you were thinking?

yliu-electra commented 2 years ago

Not sure what you meant by "completed cycles." What I visioned is that a cell is connected to the cycle for testing for a period of time of a test configuration. The cell is being tested repeatedly with such configuration. In that sense, it makes sense that the cycle number grows continuously. For what reason a cycle number should be reset?

On Thu, Jul 21, 2022 at 8:19 AM Dan Cogswell @.***> wrote:

@yliu-electra https://github.com/yliu-electra I think the key will be to only save completed cycles. That way you will know how to correct the new data when the file grows. Is that what you were thinking?

— Reply to this email directly, view it on GitHub https://github.com/Solid-Energy-Systems/NewareNDA/issues/10#issuecomment-1191545413, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZDN4TUSVH6BKKEZ4DEEYTTVVFMAPANCNFSM53LWSSJA . You are receiving this because you were mentioned.Message ID: @.***>

--

Yuxiang Liu​​ | Senior Data Scientist Email: @.*** Website: www.electravehicles.com Direct Line: +1 (970) 402‑6821 Main Line: +1 (617) 313‑7842 Address: 110 K Street, Suite 330, Boston, MA 02127 http://www.electravehicles.com/ [image: LinkedIn] https://www.linkedin.com/company/electra-vehicles-inc/ [image: Twitter] http://twitter.com/ElectraVehicles https://www.electravehicles.com/eve-ai-adaptive-controls

Yuxiang Liu | Senior Data Scientist Email: @.*** Website: www.electravehicles.com Direct Line: +1 (970) 402-6821 Main Line: +1 (617) 313-7842 Address: 110 K Street, Suite 330, Boston, MA 02127

d-cogswell commented 2 years ago

Yes, the cycle number only increases, but if you start reading from the middle of a file the cycle number will be unknown. The cycle number doesn't seem to be saved for each index.

yliu-electra commented 2 years ago

Can the index be determined reliably without reading the whole file? If so, how do you know if the cycle at that index should be the same, or the next cycle, or reset to zero (or any other possibilities?) I think those are the cases you should consider. If it is always the next cycle, then it would be easy here to accommodate that. If not, how much data do you need to read to figure it out? Would reading the previous record and checking the status name would be sufficient?

On Thu, Jul 21, 2022 at 9:07 AM Dan Cogswell @.***> wrote:

Yes, the cycle number only increases, but if you start reading from the middle of a file the cycle number will be unknown. The cycle number doesn't seem to be saved for each index.

— Reply to this email directly, view it on GitHub https://github.com/Solid-Energy-Systems/NewareNDA/issues/10#issuecomment-1191601112, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZDN4TSKXW52YYKYAE2XBFDVVFRTHANCNFSM53LWSSJA . You are receiving this because you were mentioned.Message ID: @.***>

--

Yuxiang Liu​​ | Senior Data Scientist Email: @.*** Website: www.electravehicles.com Direct Line: +1 (970) 402‑6821 Main Line: +1 (617) 313‑7842 Address: 110 K Street, Suite 330, Boston, MA 02127 http://www.electravehicles.com/ [image: LinkedIn] https://www.linkedin.com/company/electra-vehicles-inc/ [image: Twitter] http://twitter.com/ElectraVehicles https://www.electravehicles.com/eve-ai-adaptive-controls

Yuxiang Liu | Senior Data Scientist Email: @.*** Website: www.electravehicles.com Direct Line: +1 (970) 402-6821 Main Line: +1 (617) 313-7842 Address: 110 K Street, Suite 330, Boston, MA 02127