search

rdarie / Data-Analysis

Borton Lab LFP Neural Data Analysis
0 stars 0 forks source link

dataframe reshape memory leak? #17

Open rdarie opened 5 years ago

rdarie commented 5 years ago

in https://github.com/rdarie/Data-Analysis/blob/neo/dataAnalysis/preproc/ns5.py

def transposeSpikeDF(spikeDF, idxLabels, transposeToColumns):
    spikeDF.set_index(idxLabels, inplace=True)
    spikeDF = spikeDF.stack().unstack(transposeToColumns)
    idxLabels = spikeDF.index.names
    spikeDF.reset_index(inplace=True)
    return spikeDF

prf.print_memory_usage('before transposing waveforms')
spikeDF, idxLabels = transposeSpikeDF(
    spikeDF, idxLabels, transposeToColumns)
prf.print_memory_usage('after transposing waveforms')

produces

concatenating unitDF elec88#0_fr_sqrt#0
before transposing waveforms: memory usage: 4561.3 MB
after transposing waveforms: memory usage: 4863.1 MB
memory usage: 4601.8 MB
concatenating unitDF elec91#0_fr_sqrt#0
before transposing waveforms: memory usage: 4601.8 MB
after transposing waveforms: memory usage: 4903.6 MB
memory usage: 4642.2 MB
concatenating unitDF elec92#0_fr_sqrt#0
before transposing waveforms: memory usage: 4642.2 MB
after transposing waveforms: memory usage: 4944.2 MB
memory usage: 4682.7 MB
concatenating unitDF elec92#1_fr_sqrt#0
before transposing waveforms: memory usage: 4682.7 MB
after transposing waveforms: memory usage: 4984.4 MB
memory usage: 4723.1 MB
concatenating unitDF elec93#0_fr_sqrt#0
before transposing waveforms: memory usage: 4723.1 MB
after transposing waveforms: memory usage: 5025.0 MB
memory usage: 4763.7 MB
concatenating unitDF elec93#1_fr_sqrt#0
before transposing waveforms: memory usage: 4763.7 MB
after transposing waveforms: memory usage: 5065.6 MB
memory usage: 4804.3 MB
concatenating unitDF elec93#2_fr_sqrt#0
before transposing waveforms: memory usage: 4804.3 MB
after transposing waveforms: memory usage: 5106.3 MB
memory usage: 4844.8 MB
concatenating unitDF elec95#0_fr_sqrt#0
before transposing waveforms: memory usage: 4844.8 MB
after transposing waveforms: memory usage: 5146.6 MB
memory usage: 4885.2 MB
concatenating unitDF elec95#1_fr_sqrt#0
before transposing waveforms: memory usage: 4885.3 MB
after transposing waveforms: memory usage: 5187.2 MB
memory usage: 4925.8 MB
concatenating unitDF elec95#2_fr_sqrt#0
before transposing waveforms: memory usage: 4925.8 MB
after transposing waveforms: memory usage: 5227.7 MB
memory usage: 4966.4 MB
concatenating unitDF elec95#3_fr_sqrt#0
before transposing waveforms: memory usage: 4966.4 MB
after transposing waveforms: memory usage: 5268.2 MB