AttributeError: 'NoneType' object has no attribute 'dropna'

[gohingsiang]

Hello, after i try to run the run_experiments.py from /neuralnilm/scripts , it shows "AttributeError: 'NoneType' object has no attribute 'dropna'". Does anyone know how to solve it?

The version of Theano installed is (0.8.0) and Lasagne (0.1).

(my_project) ubuntu@ubuntu:~/Desktop/Goh_new/neuralnilm/scripts$ python run_experiments.py Using gpu device 0: GeForce GTX 1080 Ti (CNMeM is disabled, CuDNN 6021) /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/theano/sandbox/cuda/init.py:600: UserWarning: Your CuDNN version is more recent then Theano. If you see problems, try updating Theano or downgrading CuDNN to version 4. warnings.warn(warn) /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/theano/tensor/signal/downsample.py:6: UserWarning: downsample module has been moved to the theano.tensor.signal.pool module. "downsample module has been moved to the theano.tensor.signal.pool module.") /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/h5py/init.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._conv import register_converters as _register_converters /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/h5py/init.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import h5a, h5d, h5ds, h5f, h5fd, h5g, h5r, h5s, h5t, h5p, h5z /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/h5py/_hl/group.py:22: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .. import h5g, h5i, h5o, h5r, h5t, h5l, h5p WARNING (theano.sandbox.cuda): Ignoring call to use(1), GPU number 0 is already in use. /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/sklearn/utils/init.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .murmurhash import murmurhash3_32 /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/sklearn/utils/extmath.py:24: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._logistic_sigmoid import _log_logistic_sigmoid /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/sklearn/utils/extmath.py:26: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .sparsefuncs_fast import csr_row_norms /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/sklearn/metrics/cluster/supervised.py:23: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .expected_mutual_info_fast import expected_mutual_information /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/sklearn/metrics/pairwise.py:30: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .pairwise_fast import _chi2_kernel_fast, _sparse_manhattan Loading NILMTK activations... /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/tables/init.py:90: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .utilsextension import ( /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/tables/file.py:35: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import hdf5extension /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/tables/link.py:33: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import linkextension /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/tables/table.py:28: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import tableextension /home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/tables/index.py:33: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import indexesextension Loading kettle for UK-DALE_building_1... Loaded 2856 kettle activations from UK-DALE_building_1. Loading microwave for UK-DALE_building_1... Loaded 3181 microwave activations from UK-DALE_building_1. Loading washing machine for UK-DALE_building_1... Loaded 547 washing machine activations from UK-DALE_building_1. Loading dish washer for UK-DALE_building_1... Loaded 206 dish washer activations from UK-DALE_building_1. Loading fridge for UK-DALE_building_1... Loaded 16595 fridge activations from UK-DALE_building_1. Loading kettle for UK-DALE_building_2... Loaded 534 kettle activations from UK-DALE_building_2. Loading microwave for UK-DALE_building_2... Loaded 412 microwave activations from UK-DALE_building_2. Loading washing machine for UK-DALE_building_2... Loaded 53 washing machine activations from UK-DALE_building_2. Loading dish washer for UK-DALE_building_2... Loaded 98 dish washer activations from UK-DALE_building_2. Loading fridge for UK-DALE_building_2... Loaded 2 fridge activations from UK-DALE_building_2. Loading kettle for UK-DALE_building_3... Loaded 40 kettle activations from UK-DALE_building_3. Loading microwave for UK-DALE_building_3... UK-DALE_building_3 has no microwave. Full exception: {'instance': 1, 'type': 'microwave'} Loading washing machine for UK-DALE_building_3... UK-DALE_building_3 has no washing machine. Full exception: {'instance': 1, 'type': 'washing machine'} Loading dish washer for UK-DALE_building_3... UK-DALE_building_3 has no dish washer. Full exception: {'instance': 1, 'type': 'dish washer'} Loading fridge for UK-DALE_building_3... UK-DALE_building_3 has no fridge. Full exception: {'instance': 1, 'type': 'fridge'} Loading kettle for UK-DALE_building_4... Loaded 689 kettle activations from UK-DALE_building_4. Loading microwave for UK-DALE_building_4... Loaded 154 microwave activations from UK-DALE_building_4. Loading washing machine for UK-DALE_building_4... Loaded 154 washing machine activations from UK-DALE_building_4. Loading dish washer for UK-DALE_building_4... UK-DALE_building_4 has no dish washer. Full exception: {'instance': 1, 'type': 'dish washer'} Loading fridge for UK-DALE_building_4... Loaded 4680 fridge activations from UK-DALE_building_4. Loading kettle for UK-DALE_building_5... Loaded 173 kettle activations from UK-DALE_building_5. Loading microwave for UK-DALE_building_5... Loaded 0 microwave activations from UK-DALE_building_5. Loading washing machine for UK-DALE_building_5... Loaded 6 washing machine activations from UK-DALE_building_5. Loading dish washer for UK-DALE_building_5... Loaded 23 dish washer activations from UK-DALE_building_5. Loading fridge for UK-DALE_building_5... Loaded 1632 fridge activations from UK-DALE_building_5. Loading kettle for UK-DALE_building_2... Loaded 563 kettle activations from UK-DALE_building_2. Loading microwave for UK-DALE_building_2... Loaded 422 microwave activations from UK-DALE_building_2. Loading washing machine for UK-DALE_building_2... Loaded 57 washing machine activations from UK-DALE_building_2. Loading dish washer for UK-DALE_building_2... Loaded 105 dish washer activations from UK-DALE_building_2. Loading fridge for UK-DALE_building_2... Loaded 2 fridge activations from UK-DALE_building_2. Loading kettle for UK-DALE_building_5... Loaded 192 kettle activations from UK-DALE_building_5. Loading microwave for UK-DALE_building_5... Loaded 1 microwave activations from UK-DALE_building_5. Loading washing machine for UK-DALE_building_5... Loaded 6 washing machine activations from UK-DALE_building_5. Loading dish washer for UK-DALE_building_5... Loaded 46 dish washer activations from UK-DALE_building_5. Loading fridge for UK-DALE_building_5... Loaded 3232 fridge activations from UK-DALE_building_5. Loading kettle for UK-DALE_building_1... Loaded 0 kettle activations from UK-DALE_building_1. Loading microwave for UK-DALE_building_1... Loaded 0 microwave activations from UK-DALE_building_1. Loading washing machine for UK-DALE_building_1... Loaded 0 washing machine activations from UK-DALE_building_1. Loading dish washer for UK-DALE_building_1... Loaded 0 dish washer activations from UK-DALE_building_1. Loading fridge for UK-DALE_building_1... Loaded 0 fridge activations from UK-DALE_building_1. Loading kettle for UK-DALE_building_2... Loaded 29 kettle activations from UK-DALE_building_2. Loading microwave for UK-DALE_building_2... Loaded 10 microwave activations from UK-DALE_building_2. Loading washing machine for UK-DALE_building_2... Loaded 4 washing machine activations from UK-DALE_building_2. Loading dish washer for UK-DALE_building_2... Loaded 7 dish washer activations from UK-DALE_building_2. Loading fridge for UK-DALE_building_2... Loaded 0 fridge activations from UK-DALE_building_2. Loading kettle for UK-DALE_building_3... Loaded 20 kettle activations from UK-DALE_building_3. Loading microwave for UK-DALE_building_3... UK-DALE_building_3 has no microwave. Full exception: {'instance': 1, 'type': 'microwave'} Loading washing machine for UK-DALE_building_3... UK-DALE_building_3 has no washing machine. Full exception: {'instance': 1, 'type': 'washing machine'} Loading dish washer for UK-DALE_building_3... UK-DALE_building_3 has no dish washer. Full exception: {'instance': 1, 'type': 'dish washer'} Loading fridge for UK-DALE_building_3... UK-DALE_building_3 has no fridge. Full exception: {'instance': 1, 'type': 'fridge'} Loading kettle for UK-DALE_building_4... Loaded 44 kettle activations from UK-DALE_building_4. Loading microwave for UK-DALE_building_4... Loaded 4 microwave activations from UK-DALE_building_4. Loading washing machine for UK-DALE_building_4... Loaded 4 washing machine activations from UK-DALE_building_4. Loading dish washer for UK-DALE_building_4... UK-DALE_building_4 has no dish washer. Full exception: {'instance': 1, 'type': 'dish washer'} Loading fridge for UK-DALE_building_4... Loaded 145 fridge activations from UK-DALE_building_4. Loading kettle for UK-DALE_building_5... Loaded 19 kettle activations from UK-DALE_building_5. Loading microwave for UK-DALE_building_5... Loaded 1 microwave activations from UK-DALE_building_5. Loading washing machine for UK-DALE_building_5... Loaded 1 washing machine activations from UK-DALE_building_5. Loading dish washer for UK-DALE_building_5... Loaded 23 dish washer activations from UK-DALE_building_5. Loading fridge for UK-DALE_building_5... Loaded 1600 fridge activations from UK-DALE_building_5. Done loading NILMTK activations. Starting training for net ae, appliance washing machine. ------------- INITIALISING SyntheticAggregateSource -------------- ------------- INITIALISING RealAggregateSource -------------- Loading NILMTK mains... Loading mains for UK-DALE_building_1... Loaded mains data from building UK-DALE_building_1 for fold train from 2013-04-12 00:00:00+01:00 to 2015-01-05 06:27:12+00:00. Loading mains for UK-DALE_building_5... Loaded mains data from building UK-DALE_building_5 for fold train from 2014-06-29 17:23:42+01:00 to 2014-08-31 23:59:54+01:00. Loading mains for UK-DALE_building_1... Traceback (most recent call last): File "run_experiments.py", line 60, in main() File "run_experiments.py", line 22, in main _run_job(next_job) File "run_experiments.py", line 48, in _run_job eval("{next_job}.run('{next_job}')".format(next_job=next_job)) File "", line 1, in File "/home/ubuntu/Desktop/Goh_new/neuralnilm/experiment_definitions/e578.py", line 63, in run pipeline = get_pipeline(target_appliance, activations) File "/home/ubuntu/Desktop/Goh_new/neuralnilm/experiment_definitions/e578.py", line 185, in get_pipeline sample_period=SAMPLE_PERIOD File "build/bdist.linux-x86_64/egg/neuralnilm/data/realaggregatesource.py", line 65, in init File "build/bdist.linux-x86_64/egg/neuralnilm/data/realaggregatesource.py", line 93, in _load_mains_into_memory AttributeError: 'NoneType' object has no attribute 'dropna' Closing remaining open files:/home/ubuntu/Desktop/Goh/Data/ukdale.h5...done

[Jacoobr]

Hello, gohingsiang. I met the same problem too. Have you fix the problem?

[gohingsiang]

Hi, Jacoobr, I still struggle with it, however I suspect this problem is due to the version of Lasagne or Theano that we used. Therefore I did ask Jack Kelly regarding the version of Lasagne and Theano that he used that time but unfortunately he don't remember about it. Then, after that i try to use Theano 0.6.0 and Theano 0.7.0 I'm still getting the same error.

[Jacoobr]

Hi, gohingsiang, I think this may caused by pandas version, so I try to down my pandas version from 0.22 to 0.19.2. But I still get this 'dropna' error. By the way, I also run a similar Energy Disaggregation project that comes from hy10327's project: https://github.com/hyl0327/neuralnilmtp . I get the same error too. I hope you can try hy10327's project too, may be the code work nicely for you. The problem confused me many days. What's more important, Do you get the dataset from UK-DALE : http://data.ukedc.rl.ac.uk/simplebrowse/edc/efficiency/residential/EnergyConsumption/Domestic/UK-DALE-2015/UK-DALE-disaggregated/ukdale.h5.tgz ?

[gohingsiang]

Thanks for the information! I will let you know the result after i tried. And yes that is where i get the dataset from UK-DALE and currently I'm using the file extracted from ukdale.h5.tgz.

[Jacoobr]

Thank you, gohingsiang. May i see your code in the 'neuralnilm/experiment_definitions/e578.py"' script file? Have you edit the author's code?

[gohingsiang]

Following are the code for the neuralnilm/experiment_definitions/e578.py that I used. I doesn't change the code inside it except for the NILMTK_FILENAME which has been changed to NILMTK_FILENAME = '/home/ubuntu/Desktop/Goh/Data/ukdale.h5'

!/usr/bin/env python0

from future import print_function, division from os.path import join import sys from neuralnilm.config import config

import logging logger = logging.getLogger('neuralnilm')

experiment_definition_path = config.get('Paths', 'experiment_definitions') job_list_filename = join(experiment_definition_path, 'job_list.txt')

def main(): if experiment_definition_path not in sys.path: sys.path.append(sc experiment_definition_path)

sys.path.insert(0, experiment_definition_path) #original

testing = sys.path.insert(0, experiment_definition_path)#testing
print ("\nthis is testing ",testing)
print ("\nthis is sys.path ",sys.path) 
next_job = _get_next_job()

while next_job:
    try:
        _run_job(next_job)

    except KeyboardInterrupt:
        delete_this_job = raw_input(
            "Delete this job from job list [Y/n]? ").lower()
        if delete_this_job != "n":
            _delete_completed_job()
        continue_running = raw_input(
            "Continue running other experiments [N/y]? ").lower()
        if continue_running != "y":
            break
    else:
        _delete_completed_job()
        next_job = _get_next_job()

def _get_next_job(): with open(job_list_filename, 'r') as fh: next_job = fh.readline().strip() return next_job

def _run_job(next_job): logger.info("Running {}".format(next_job)) exec("import {:s}".format(next_job)) print ("\nthis is import{:s} \n","import {:s}") from neuralnilm.utils import configure_logger configure_logger('neuralnilm.log') eval("{next_job}.run('{next_job}')".format(next_job=next_job))

def _delete_completed_job(): with open(job_list_filename, 'r') as fh: remaining_jobs = fh.readlines()[1:] with open(job_list_filename, 'w') as fh: fh.writelines(remaining_jobs) logger.info("Remaining jobs = {}".format(remaining_jobs))

if name == "main": main()

[gohingsiang]

Hi Jacoobr, after I try it with the pandas 0.19.2 I getting the same error as you mentioned before.

ValueError("cannot reindex a non-unique index " ValueError: cannot reindex a non-unique index with a method or limit

[Jacoobr]

Thank you gohingsiang, Can you tell me which pandas version that you use before you test with the version 0.19.2?

[gohingsiang]

I was using pandas 0.16.2 before due to the suggestion from https://github.com/JackKelly/neuralnilm/issues/1.

Beside from 0.16.2 and 0.19.2 as you suggested, I also tried pandas 0.16.0 and 0.17.1. But unfortunately I'm still getting the AttributeError: 'NoneType' object has no attribute 'dropna'.

I also tried different version of Theano such as 0.6.0, 0.7.0 and 0.8.0 but nothing jus work.

[gohingsiang]

By the way, I tried to run the code Neural NILM test.ipynb and NeuralNILM design.ipynb from the neuralnilm/notebooks/ and it's working, but the structure of the network is not the same as writen in Jack Kelly paper

[Jacoobr]

Thank you very much, gohingsiang. Your message help me a lot. Now i try to go back to Jack Kelly's paper and his code to see what exactly he did in his project. Any progress about this AttributeError: 'NoneType' object has no attribute 'dropna', i will tell you. Thank you .

[gohingsiang]

My pleasure. Any progress regarding AttributeError: 'NoneType' object has no attribute 'dropna', i will inform you as well. Thanks in advance!

By the way, the result of the paper is written based on e567.py from his previous github Neural NILM Prototype. I tried to run the code but I'm gettng another error regarding the keyword argument for "border_mode". Maybe you can try on it.

[Jacoobr]

Thank you. I will try the experiment based on e567.py later. Then, i will tell you the result of mine. By the way, i faced a new error 'NameError: name 'Plotter' is not defined'. I guess this is the version of my matplotlib 1.4.3 caused. Can you tell me what's your matplotlib's version that you used? Thank you.

[Jacoobr]

Hi, gohingsiang. Would you mind help me see which package the Plotter() dependency. In the code neuralnilm-master/notebooks/NeuralNILM design.ipynb `plotter = Plotter() # this is the code that caused NameError:name 'Plotter' is not defined disaggregator = Disaggregator()

trainer = Trainer( net, data_pipeline=None, repeat_callbacks=[ ( 10, Trainer.validate), (1000, plotter.plot), (1000, disaggregator.disaggregate), (1000, disag_metrics.run_metrics) ] ) ` Thank you.

[gohingsiang]

I removed this part of the code and I'm getting the result, I think that this part of the code is one of the code that Jack Kelly develop at that moment, but the code work fine even if you removed this part of the code.

[Jacoobr]

Thanks for your reply. The code works just like you said.

[gohingsiang]

But please take note that the structure of the network used is not the same as in Jack Kelly's paper. I was thinking to change the coding in the ipynb but i notice that they way how it handle the data is slightly different with the e567.py and e578.py.

[Jacoobr]

Hi, gohingsiang. Do you have any progress about training the net? When i try to run the script of e567.py, an error raised like this: OSError: [Errno 2] No such file or directory: '/data/dk3810/figures/e567_microwave_ae'. Do you have any idea about this error? By the way , i find nothing about the 'e567_microwave_ae' in Jack Kelly's previous project.

[gohingsiang]

Hi, Jacoobr, for the e567.py, I was facing some error regarding keyword argument for the boder_mode, did you face this problem before?

[Jacoobr]

Hello, gohingsiang, i was so busy that i didn't reply your question yesterday. I'm sorry about that. For your question, i also faced this error by running the script e567.py. It raised an error the keyword argument for "border_mode" like you said. I fixed this error by annotating the code 'border_mode' in the script e567.py temporary. Then i ran the script again, this time it worked. I see the net begin to train with many out put like this: ` .......Running net.fit for e567_microwave_rectangles Starting training for 3000 iterations.

Update	Train cost	Valid cost	Train / Val	Secs per update

Saving plots... /usr/local/anaconda2.4.2b/python2.7/site-packages/matplotlib/collections.py:590

: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison if self._edgecolors == str('face'): Saving params... Saving activations... Finished saving. 0 | 0.220820 | 0.194270 | 1.136669 | 63.896s 1 | 0.317625 | 0.194270 | 1.634969 | 0.989s 2 | 0.217491 | 0.194270 | 1.119533 | 0.987s 3 | 0.109623 | 0.194270 | 0.564285 | 1.005s 4 | 0.258685 | 0.194270 | 1.331575 | 0.990s 5 | 0.159591 | 0.194270 | 0.821490 | 1.050s ......` By the way, i use Jack Kelly's neuralnilm_prototype project for this test. Hope this can help you.

[gohingsiang]

Hi, Jacoobr, I tried to replace the word "border_mode" with "pad" inside the coding after I referred http://lasagne.readthedocs.io/en/latest/modules/layers/conv.html#lasagne.layers.Conv1DLayer, however I'm getting another error.

Exception: ('The following error happened while compiling the node', GpuDnnConvDesc{border_mode='full', subsample=(1, 1), conv_mode='conv', precision='float32'}(MakeVector{dtype='int64'}.0, MakeVector{dtype='int64'}.0), '\n', 'nvcc return status', 2, 'for cmd', '/usr/local/cuda-8.0/bin/nvcc -shared -O3 -arch=sm_61 -m64 -Xcompiler -fno-math-errno,-Wno-unused-label,-Wno-unused-variable,-Wno-write-strings,-DCUDA_NDARRAY_CUH=c72d035fdf91890f3b36710688069b2e,-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,-fPIC,-fvisibility=hidden -Xlinker -rpath,/home/ubuntu/.theano/compiledir_Linux-4.13--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-2.7.12-64/cuda_ndarray -I/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/theano/sandbox/cuda -I/usr/local/cuda-8.0/include -I/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/numpy/core/include -I/usr/include/python2.7 -I/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/theano/gof -o /home/ubuntu/.theano/compiledir_Linux-4.13--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-2.7.12-64/tmppHh_Rr/358ed08bfb8a798e5d73c5258b893f50.so mod.cu -L/usr/lib -lcudnn -lpython2.7 -lcudart', "[GpuDnnConvDesc{border_mode='full', subsample=(1, 1), conv_mode='conv', precision='float32'}(<TensorType(int64, vector)>, <TensorType(int64, vector)>)]")

May I have a look on your code that annotating the code 'border_mode' in the script e567.py temporary?

[Jacoobr]

Hi, gohingsiang. The code that i annotated in the script e567.py like this:
287 # 'border_mode': 'valid' ...... 295 #'border_mode': 'valid' ...... 374 #'border_mode': 'same' ...... 443 #'border_mode': 'valid' ...... 478 #'border_mode': 'full'
But please take note that this is only a temporary way to run the code. In my case, i tried to run the three nets that Jack Kelly said in his paper.But i only ran the rectangle net successfully. The other two nets got a 'reshape error'. I guess this error was caused by the five lines code that i annotated in the script e567.py.

[gohingsiang]

Hi, Jacoobr, maybe you can try to replace the 'border_mode': 'valid' with 'pad': 'valid'.

            'type': Conv1DLayer,  # convolve over the time axis
            'num_filters': 16,
            'filter_size': 4,
            'stride': 1,
            'nonlinearity': None,
            'pad': 'valid', #addon
    #'border_mode': 'valid' #original code

It works for me and hopefully it works for you, but right now I was trying to find the code for the metric but I'm uncertain about which code i should used in Neural NILM Prototype. Do you have any idea?

[Jacoobr]

Hi gohingsiang, thanks for your suggestion.I will try it later. What's more, maybe you can find something ahout the metric from Where the code send the metric to MongDB in Jack Kelly's project. There are some message about the metric as follow in his README.md : `trainer.py includes a Trainer class which trains each net and sends metrics to the Mongo database.

neuralnilm/neuralnilm/data includes the mechanisms for loading and transforming data. This is designed to be very modular so you can easily mix and match data sources, pre-processing steps and networks. For a quick overview of how this all works, take a look at some of the python files in the experiment_definitions directory.

neuralnilm/neuralnilm/monitor contains the mechanisms for loading metrics and metadata from the Mongo database and plotting it.` Hope this can help you.

[gohingsiang]

Hi, Jacoobr, thanks for your information, in fact now I'm trying to modified the code for the e578.py especially in real_agg_source in line 179. By the way, I can't found any folder regarding the scores or params after i run the code for e567.py, did you get any file regarding it?

[Jacoobr]

Did you mean where the output stored by running the script e567.py? When i finished running the script, i can see the output of the params and many output *.pdf file in the file where i set it in e567.py like this: `NAME = os.path.splitext(os.path.split(main.file)[1])[0]

PATH = "/homes/dk3810/workspace/python/neuralnilm/figures"

PATH = "/home/jacoob/workStation/extraProject/figures/" #the path that the output stored

PATH = "/home/jack/experiments/neuralnilm/figures"

UKDALE_FILENAME = '/home/jacoob/workStation/extraProject/dataSet/ukdale.h5' `

[gohingsiang]

I did found those figures and files, but when I run the code in the Neural NILM test.ipynb and NeuralNILM design.ipynb from the neuralnilm/notebooks/, I found there is a file name net_params.h5 in the same folder with the figures, however, I couldn't found any file in that particular folder after running the e567.py code.

[Jacoobr]

Hello gohingsiang, Did you mean where the net params stored when the e567.py finished ? I can see the output file like : e567_microwave_rnn.hdf5，e567_microwave_rectangles.hdf5　,etc. Maybe JackKelly changed the file format which he used in his neuralnilm project.And i open the net_params.h5 file and e567_microwave_rnn.hdf5(the file that output by running e567.py script.) file with hdfview software. I can see the same data format.So i guess the *.hdf5 is the file where params stored.

[gohingsiang]

Okay, thanks for your information.

[Jacoobr]

Hi goingsiang, Have you test the script disag_567.py in Jack Kelly's neuralnilm-prototype project. I try to test this script and i get an error like this: return np.concatenate((np.zeros(padshape, dtype=arr.dtype), arr), MemoryError Exception TypeError: TypeError("'NoneType' object is not callable",) in <function _remove at 0x7f0dab849758> ignored Do you have any idea about this error?

[gohingsiang]

Hi Jacoobr, after running the code I get this following error:

Loading params from /home/ubuntu/Desktop/Goh_new/Data/e567_output/e567_modified_washing machine_rectangles/e567_modified_washing machine_rectangles.hdf5... Traceback (most recent call last): File "disag_567.py", line 280, in neural_nilm_disag() File "disag_567.py", line 263, in neural_nilm_disag net = get_net(appliance, architecture) File "disag_567.py", line 131, in get_net path=join(NET_BASE_PATH, experiment_name)) File "/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/NeuralNILM-0.0.1-py2.7.egg/neuralnilm/net.py", line 515, in load_params epoch_group = f[epoch_name] File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/h5py/_hl/group.py", line 167, in getitem oid = h5o.open(self.id, self._e(name), lapl=self._lapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5o.pyx", line 190, in h5py.h5o.open KeyError: "Unable to open object (object 'epoch300000' doesn't exist)"

I check the content inside the e567_modified_kettle_ae.hdf5 but inside only content epoch000000, it seem like it doesn't save the param for my case. Did you face this problem before?

[Jacoobr]

I faced this problem before? There is only epoch000000 in the .hdf5 file like you said. I guess this may caused by the iteration(epochs) number(3000) that i set when i training the net. So i changed it to 0, it means let she script only load epoch000000. Then, this problem gone.Hope the message can help you.

[gohingsiang]

I tried to KeyboardInterrupt during the training, then it came out with a menu, then i choice the Save plots and params, then i was able to obtain the epoch000000 and epoch003100 inside the hdf file, I think that it's due to the method for saving the params. I think that we should change the way to save the params as in /neuralnilm_prototype/neuralnilm/experiment.py (show as below). I'm afraid the data from epoch000000 might not be the params obtained from the last training.

def menu(net, epochs):

Print menu

print("")
print("------------------ OPTIONS ------------------")
print("d: Enter debugger.")
print("s: Save plots and params.")
print("q: Quit all experiments.")
print("e: Change number of epochs to train this net (currently {})."
      .format(epochs))
print("c: Continue training.")
print("")

# Get input
selection_str = raw_input("Please enter one or more letters: ")

# Handle input
for selection in selection_str:
    if selection == 'd':
        import ipdb
        ipdb.set_trace()
    elif selection == 's':
        net.save()
    elif selection == 'q':
        sure = raw_input("Are you sure you want to quit [Y/n]? ")
        if sure.lower() != 'n':
            raise
    elif selection == 'e':
        new_epochs = raw_input("New number of epochs (or 'None'): ")
        if new_epochs == 'None':
            epochs = None
        else:
            try:
                epochs = int(new_epochs)
            except:
                print("'{}' not an integer!".format(new_epochs))
    elif selection == 'c':
        break
    else:
        print("Selection '{}' not recognised!".format(selection))
        break
print("Continuing training for {} epochs...".format(epochs))
fit(net, epochs)

[Jacoobr]

Hello,gohingsiang. You are right. The number of iteration that you set when training the net should be bigger than that i set 3000. The more iteration we training, the better feature we will get when we disaggregate a single application from aggregate data.Now i'm trying to disaggregate one application like kettle, washing machine use the script disag_567.py.But when the script load the main data from building_{:d}_mains.csv, i get MemoryError like i said before. Maybe the code works for your test. By the way, i use 100000 rows data(the data i stored in the building_1_mains.csv) of the mains.data that i get from UK-DALE dataset to test the script.

[gohingsiang]

May i know where is your location of the building_1_mains.csv aforementioned? Because i couldn't found any related .csv file. Is that the file generated from the disag_567.py?

[gohingsiang]

Hi Jacoobr, may I know where you get the ground_truth_and_mains?

[Jacoobr]

I get the file that in the ground_truth_and_mains file are as follow:: building_1_mains.csv、building_2_mains.csv、building_5_mains.csv These 'building_i_mains.csv' files made by 'mains.dat' that come from house1, house2, house5 in dataset.Down the end of the page, you will see the mains.dat data. I get the .csv file by Excel, so the .csv file only have 1048576 rows. Because the lines of mains.dat file over the limit of Excel can handled.

[gohingsiang]

Maybe you can try to use the ground_truth_and_mains from http://jack-kelly.com/neuralnilm/ and I was able to run without error, I hope that this will able to solve your MemoryError.

[Jacoobr]

Hi, gohingsiang. Thank you very much.I just forget the site that you mentioned.Thank you.

[Jacoobr]

Hello, gohingsiang.Hope the message as follows can help you fix the problem that you mentioned yesterday. some latex package that i installed in my Ubuntu16.04. And this works for my to run the BuildSys_results_plot.ipynb nicely. Hope this can help you.

[gohingsiang]

Thanks for you reply, I did found this post and the error has been solved! Thanks for all the information!

[gohingsiang]

Hi Jacoobr, may I have your email in case there something confusing, we can discuss individually.

[Jacoobr]

Hi gohingsiang, you can contact me with this email: .I's my pleasure to discuss problem with you.

[changfangyi]

Do u guys use ukdale.h5 for your project ? u can try redd.h5 and https://github.com/hyl0327/neuralnilmtp is designed for redd actually.

[gohingsiang]

Yes, changfangyi, we are using the ukdale.h5 for this project. But for me, I think for this particular moment I'm gonna put more attention on UK-DALE for this project. Anyway thanks for the information shared. I really appreciated it!

[changfangyi]

I wish that your problem was solved. I use mac or linux to perform my project, and I don't face the bug "AttributeError: 'NoneType' object has no attribute 'dropna'", even using ukdale.

[gohingsiang]

Hi, @changfangyi ,may i know your reproduced project is based on which github? Is it the neuralnilm or the neuralnilm_prototype?

[changfangyi]

My project is based on neuralnilm. Here is the code: https://github.com/changfangyi/III_NILM_DeepLearning

[Jacoobr]

Hi, @changfangyi , Can you provide me with REDD dataset for the future work with NILM project? I can't download the REDD dataset from the site that you mentioned for some reasons. Thank you very much.

[changfangyi]

ok, you can download the data from my one drive

https://1drv.ms/f/s!AkX-2oVmhJj3gbc3dM3EFlIgblmbVw