creating ABF1 files from scratch

swharden commented 5 years ago

A lot of people seem to use software like MiniAnalysis which expects data to arrive as ABF1 files. It may be useful to create a pyABF module which allows the creation of ABF1 files from scratch.

The goal of this ticket is to track progress toward exporting data as ABF1 files that MiniAnalysis can import. In the process a sweep-synthesis module will be created to create simulated data which contains events suitable for detection.

Tasks

[x] code to synthesize whole-cell patch-clamp data from scratch
- [x] EPSCs and IPSCs (exponential)
- [x] EPSPs and IPSPs (alpha)
- [x] action potentials
- [x] gaussian amplifier noise
- [x] cell drift (wobble) instability
- [x] integrate trace synthesis code into pyABF as a module
- [x] document trace synthesis
[x] ABF1 file export
- [x] suitable for ClampFit to open
- [x] suitable for MiniAnalysis to open
- [x] integrate ABF1 export into pyABF
- [x] document ABF1 export

jeakwon commented 5 years ago

I am working at the lab of Dr. C.Justin. Lee, the developer of the MiniAnalysis. If you need any bridge or detailed information about it, I can discuss with him. Thank you for all your efforts

swharden commented 5 years ago

Hi @jeakwon, thank you for your offer! I have never used MiniAnalysis before, but I have seen it cited in the literature and heard positive things about it. I will first work toward making pyABF export ABF1 files that ClampFit can open, and once that is working I may ask for your assistance to verify that those ABFs can be properly opened by MiniAnalysis.

If you would prefer to correspond privately, you are welcome to email me any time (SWHarden@gmail.com). Thanks again!

swharden commented 5 years ago

Once these features are fully working they will be incorporated into pyABF. Until then they will be worked on in the /dev/python folder. The ABF files I'm working with (the ones generated from scratch) are also in that folder.

ABF de novo synthesis

I started some code to generate realistic whole-cell patch-camp recordings from scratch. I decided to support noise, filtering, and creation of events to produce something interesting to look at (and practice event detection). I intend to support action potential generation, but for now only EPSPs, IPSPs, EPSCs, and IPSCs are supported. This code lives in 2018-11-24 simulated data.py and I tested it in the following way:

I made a dataset of ten 5-second sweeps sampled at 20 kHz
10 Hz IPSCs (monoexponential) were stochastically placed in each sweep
The time constant of each is about 100ms, but randomly varies slightly
Event amplitude is random (not normally distributed though)
Gaussian noise was added to the trace to simulate amplifier noise
A slight wobble was added to the trace to reflect real recording conditions
The signal was low-pass filtered (due to Ra, not the hardware filter)
The signal was visualized from within Python (left)
The signal was saved as an ABF1 file and visualized with ClampFit (right)

The python script I used to convert the signal to an ABF1 file is 2018-11-25 abf1 de novo.py.

From-scratch data in Python/Matplotlib	Exported ABF1 file viewed in ClampFit

swharden commented 5 years ago

After synthesizing ABF1 files with integer notation (https://github.com/swharden/pyABF/commit/19956070018bbd4942313c5c5124de71d82dff1b), MiniAnalysis now loads them. It seems episodic files are fine and gap-free files are not required. Currently the vertical scaling is wrong, but that's a coding problem on my end. Once fixed, MiniAnalysis should be supported.

swharden commented 5 years ago

ABF Import into MiniAnalysis

I further modified the ABF1-writing python script to support integer notation instead of float. The new code (2018-11-25 abf1 de novo.py) generates an ABF1 file from scratch which can be imported directly into MiniAnalysis.

MiniAnalysis	ClampFit

@jeakwon based on your familiarity with this program, do you think this is sufficient to conclude MiniAnalysis fully supports analysis of these files? Since I only have a demo version of MiniAnalysis, I'd appreciate your input. The file I generated from scratch is 2018-11-25 abf1 de novo.abf and you can download it and test it out with your program. Your feedback would be greatly appreciated!

jeakwon commented 5 years ago

Sorry for late response. I will keep eye on this issue as frequent as I can. I will test your code from today and try to give you any feedback if there exist. But at a glance, it seems working fine.
I'll come back to this issue again. Thank you for your effort!

jeakwon commented 5 years ago

@swharden I have forked on your pyABF and ran 2018-11-25 abf1 de novo.py. As you have shown in the MiniAnalysis png file and ClampFit png file, I was able to open de novo.abf with MiniAnalysis and successfully detected events with MiniAnalysis. Until now, I have been converting abf2 file with clampfit save as function with specific option, "ABF 1.8 (integer) (*.abf)". With this way, it is possible to open abf2 file in MiniAnalysis. I am not sure pyabf de novo synthesis method is almost similar (or even better) compared to this method. Anyways, I think clampfit based method can be a criterion for converting option since clampfit based converting was successful for analyzing events. I'll notice if there is any updates on this issue

swharden commented 5 years ago

@jeakwon thanks for looking into it! I'm glad to hear you were able to successfully perform event detection in MiniAnalysis on the ABF1 files generated with this code. I will soon add a "saveABF1" feature to the pyABF library. This will allow any ABF loaded with pyABF to be saved as a MiniAnalysis-compatible ABF1 file using code similar to this:

import pyabf
abf = pyabf.ABF("input_abf2.abf")
abf.saveABF1("output_abf1.abf")

Batch conversion of a folder of ABF2 files to ABF1 files can then be performed with:

import pyabf
import glob
for filename in glob.glob("/path/*.abf"):
    abf = pyabf.ABF(filename)
    abf.saveABF1(filename+"_abf1.abf")

I'll update this ticket when this new feature is ready to test! Thanks again for your input.

jeakwon commented 5 years ago

I really appreciate your effort, and I am ready to test your above functions with my own recording files. If it works fine, I am planning to make simple distributable gui based exe program for our lab members and other potential users of MiniAnalysis, with your codes if you allow me to do it. Thank you.

swharden commented 5 years ago

I really appreciate your effort, and I am ready to test your above functions with my own recording files. If it works fine, I am planning to make simple distributable gui based exe program for our lab members and other potential users of MiniAnalysis, with your codes if you allow me to do it. Thank you.

@jeakwon I'm happy that pyABF will soon support MiniAnalysis-compatible ABF exporting! The new code should be ready in the next few days, and I'll post here when it's ready for you to test. You are welcome to use pyABF in any way you like, including building a GUI for it or integrating it with commercial software like MiniAnalysis.

swharden commented 5 years ago

@jeakwon pyABF now supports saving any ABF file as an ABF1 file compatible with MiniAnalysis.

I issued a release (2.0.30) which is now live on PyPi:

pip install --upgrade pyabf

You can load any ABF file and save it as an ABF1 file like this:

import pyabf
abf = pyabf.ABF("input.abf") 
abf.saveABF1("output.abf")

Data values may be slightly different due to the way pyabf and the ABF header handle 16-bit ADCs and the conversion between floating point data (in memory) and integer data (in the file), but these differences should be extremely small (<0.1 pA or mV).

I look forward to hearing if this new update works for you!

ViraTsintsadze commented 5 years ago

@swharden thank you so much for this upgrate! I'm excited to integrate it in my code. So far, the code below gives an error

import pyabf 
abf = pyabf.ABF("f1.abf")
abf.saveABF1("output.abf")

File "C:\Users\Vira\Anaconda3\lib\site-packages\pyabf\abfWriter.py", line 76, in writeABF1
    struct.pack_into('h', data, bytePosition, int(value*valueScale))
error: short format requires (-32768) <= number <= 32767

I tryed to save my f1.abf file with Clampfit in 4 different options (floating/integer, 1.8 and modern) and the error is same. Here's my f1.abf file: f1.zip

swharden commented 5 years ago

Hi @ViraTsintsadze thanks for the update! I know what's causing this problem and will have it corrected soon. I opened a new ticket to deal with this issue. Thanks for sending your ABF! I'll make sure it gets supported soon.

The new issue tracking this bug is: https://github.com/swharden/pyABF/issues/65

swharden commented 5 years ago

@ViraTsintsadze thank you again for your demo file. It was very useful to test with. This issue has been resolved, and is now live on pypi:

pip install --upgrade pyabf

Let me know if you have any trouble! Best, Scott

ViraTsintsadze commented 5 years ago

It works! Awesome, thank you so much, Scott

swharden commented 4 years ago

@jeakwon I made a new Windows program (.exe) which may be of interest to you. It converts ABF files (ABF1 and ABF2) to ATF or CSV text file format. Would this be useful for your MiniAnalysis program? It's a little crude at the moment (just a console application) but it gets the job done, and it would be easy to extend with a GUI if it proves useful.

https://github.com/swharden/AbfConvert

jeakwon commented 4 years ago

Thanks to know, I will try out soon, but seems really useful. I really want our lab members take advantage of your program, though many people working on with abf files usually not familiar to command line method. so do you have any plan to build gui? by the way, I also liked your way of using official dll provided by company.

swharden commented 4 years ago

I really want our lab members take advantage of your program, though many people working on with abf files usually not familiar to command line method. so do you have any plan to build gui?

I guess a command line can be intimidating for people who are less familiar with computers 🤔

I made a GUI to help. If you try it out I'm interested in what you think! https://swharden.com/software/AbfConvert/

jeakwon commented 4 years ago

Thank you, I and my colleague tried out your gui abf convert.

With out any help, he used it properly, and liked it.

By the way, your above link https://swharden.com/software/abfConvert/ does not work because of "a"bfConvert. it works with "A"bfConvert

https://swharden.com/software/AbfConvert/

That made me to figure out for one~two days but I finally made it. Anyways, thank you for your service on E-phys tools including pyabf lib. I am a big fan and active user of it.

swharden commented 4 years ago

Hey @jeakwon, I'm happy to hear you found it useful! I changed the capitalization of the URL after posting here and forgot to come back and update the link 😅 I'm happy you figured it out though.

https://swharden.com/software/AbfConvert/

Best, Scott

PikaPei commented 1 year ago

Hi Scott,

Thanks for the wonderful tool! I just wonder if abfWriter can write sweeps with different length?

swharden commented 1 year ago

Hi @PikaPei, I'm happy you found this useful!

Presently this code only generates fixed-length sweeps.

Actually, since this generates ABF1 format files, it may be possible that ABF1 format does not support sweeps with different lengths too.

If you decide to look into it further, find it is possible, and create some code to do it, let me know and I can include it in a future release of pyabf!

Thanks, and good luck! 🚀 Scott

PikaPei commented 1 year ago

I'll stick to ABF1 format first. Thanks for the kind explanation!

Best, Pei

swharden / pyABF