Open anasvaf opened 5 years ago
Hi
Im having the same error, so i went to the train_across_buildings
function and played a bit with it.
I found that the error pointed to the same line (117) no matter what was there. At first forcing self.mmax = x
and then even leaving a white line there or putting other lines of code there.
This lead me to think that the pandas error is not related with the function, or that part of the function, but for some reason it always points there. Im not certain of anything anyways :(
Let me know if the same happens to you!
Hello Tomas,
I have also tried to play with the train across buildings. Here is how I tried to "debug" the function
if self.mmax == None:
print(mainchunks)
for m in mainchunks:
print(len(m))
input("wait")
If you look at the Series that are created, you will notice for house 3, if I am not mistaken, you get an empty dataframe, where you cannot calculate the maximum value. I believe that is the core of the error.
Let me know if you get something similar.
To be more specific, here is what you get when you calculate inside the for loop m.max()
Hmm Im using the REDD database, so i haven't checked, but maybe it's just that the third house of ukdale doesn't have a kettle meter. Check it and let me know, because i used to have that error and now i don't have it anymore and don't know why π .
Nonetheless, that's not my case (i don't have an empty dataframe) and i'm still having problems.
I changed the code as u suggested to
if self.mmax == None:
print(mainchunks)
for m in mainchunks:
print(len(m))
input("wait")
self.mmax = max([m.max() for m in mainchunks])
and training with three houses i got
23179
wait
31616
wait
56886
wait
For some reason my code is now running a bit past that, it entered into the train_across_buildings_chunk
method and died in doing the random.shuffle(batch_indexes)
.
I fixed that modifying the line above that from batch_indexes = range(min(num_of_batches))
to batch_indexes = list(range(min(num_of_batches)))
But now the reshape X = np.reshape(mainpart, (batch_size, self.window_size, 1))
raises me this error:
Traceback (most recent call last):
File "redd-test.py", line 39, in <module>
disaggregator.train_across_buildings(train_mains, train_meters, epochs=1, sample_period=sample_period)
File "/Users/TSV/Desktop/Progra/IPre/Server/seq2seq/2buildings/shortseq2pointdisaggregator.py", line 151, in train_across_buildings
self.train_across_buildings_chunk(mainchunks, meterchunks, epochs, batch_size)
File "/Users/TSV/Desktop/Progra/IPre/Server/seq2seq/2buildings/shortseq2pointdisaggregator.py", line 212, in train_across_buildings_chunk
X = np.reshape(mainpart, (batch_size, self.window_size, 1))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 279, in reshape
return _wrapfunc(a, 'reshape', newshape, order=order)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 51, in _wrapfunc
return getattr(obj, method)(*args, **kwds)
ValueError: cannot reshape array of size 2100 into shape (42,100,1)
Closing remaining open files:/Users/TSV/Desktop/Progra/IPre/data/REDD/redd.h5...done/Users/TSV/Desktop/Progra/IPre/data/REDD/redd.h5...done
Let me know if you get here or if you don't get rid of the pandas error.
Hello everybody and sorry for the late reply. Which dataset are you using? For UKDALE, building 3 there is no data for the specified date range
train.set_window(start="13-4-2013", end="1-1-2014")
This may explain the empty dataframe.
Hi Ody!
First of all thanks a lot for sharing your work and maintaining support! It's been really helpful π
I'm using REDD and don't have the empty dataframe problem, but another error as i explained above.
Does the train_across_buildings
method as it is in the repo work fine for you?
Hello guys, I managed to fix the error by simply removing the train and test.set_window. I assume that the toolkit can identify the common dates within all the buildings. I am using the UK-DALE dataset for my experiments.
I print the batch 0 array for my 4 training houses and it is the following: Batch 0 of [1185228, 317437, 70848, 366821]
The error that I get know is that my data has to be 1-D.
Any hints for this one?
I managed to fix it by changing the for loop inside the train_across_buildings_chunk as follows:
# Create a batch out of data from all buildings
for i in range(num_meters):
mainpart = mainchunks[i]
meterpart = meterchunks[i]
mainpart = mainpart[b*batch_size:(b+1)*batch_size]
meterpart = meterpart[b*batch_size:(b+1)*batch_size]
X = np.reshape(mainpart.values, (batch_size, 1, 1))
Y = np.reshape(meterpart.values, (batch_size, 1))
X_batch[i*batch_size:(i+1)*batch_size] = np.array(X)
Y_batch[i*batch_size:(i+1)*batch_size] = np.array(Y)
from the pandas Series, we needed only the values in order to reshape.
Tomas if you just change the X = np.reshape(mainpart, (batch_size, self.window_size, 1))
to X = np.reshape(mainpart.values, (batch_size, self.window_size, 1))
the code should work.
Now the script is iterating over the batches. I will let you know regarding the progress.
I managed to fix the error by simply removing the train and test.set_window. I assume that the toolkit can identify the common dates within all the buildings. I am using the UK-DALE dataset for my experiments.
I am not sure whether the toolkit detects common sections within building. Also by removing the limit, you are now getting all of the data available for each building.
@TomasSalvadores Your problem seems different than the one mentioned by the OP, please open a new issue and describe your problem in order to be able to discuss it.
I managed to fix the error by simply removing the train and test.set_window. I assume that the toolkit can identify the common dates within all the buildings. I am using the UK-DALE dataset for my experiments.
I am not sure whether the toolkit detects common sections within building. Also by removing the limit, you are now getting all of the data available for each building.
You are right. I am just using all of the data available for each building. Sorry for any confusion :)
@anasvaf Thanks, you saved my day! Although I think there is an error in your code, I had to change
Y = np.reshape(meterpart.values, (batch_size, 1))
to
Y = np.reshape(meterpart.values, (batch_size, 1, 1))
in order to make it work.
Hi @OdysseasKr, @anasvaf, @maechler, @TomasSalvadores
I have experimented using dae.train_across_buildings
and got the error as well. I tested with UK-DALE dataset and tried to learn fridge model from House 1 and House 2. I tried fixing follow the recommendations by both @anasvaf and @maechler but they still didn't work. Below was the error (the same for both methods).
========== TRAIN ============ CHECKPOINT 0 0 Batch 0 of [25, 25]
AttributeError Traceback (most recent call last) ~\Anaconda3\envs\nilmtk-env\lib\site-packages\numpy\core\fromnumeric.py in _wrapfunc(obj, method, *args, *kwds) 55 try: ---> 56 return getattr(obj, method)(args, **kwds) 57
~\Anaconda3\envs\nilmtk-env\lib\site-packages\pandas\core\generic.py in getattr(self, name) 5066 return self[name] -> 5067 return object.getattribute(self, name) 5068
AttributeError: 'Series' object has no attribute 'reshape'
During handling of the above exception, another exception occurred:
ValueError Traceback (most recent call last)
Hi Ody! First of all thanks a lot for sharing your work and maintaining support! It's been really helpful π I'm using REDD and don't have the empty dataframe problem, but another error as i explained above. Does the
train_across_buildings
method as it is in the repo work fine for you?
where can i download redd dataset?
Hello Odyssea,
I am trying to replicate the results based on the Kelly et al. paper. Using your code I changed the train building to the following:
and then I call the train_across_buildings
I am creating lists for the 4 houses that I need during the training but I get the following error when I run the RNN for the function "train_across_buildings"
Could you give me some hint on how to change the code?
Best, Tasos