Error training with FHMM (unable to allocate 26.0GB for an array)

[crn565]

I am training a new dataset similar a IAWE with FHMM. The train works if I used only 5 appliances but If I use 10 appliances show the error. This is the code:

from nilmtk.legacy.disaggregate import FHMM num_states_dict ={'Electric furnace':2, 'Microwave':2,'Television':2,'Kettle':2,'Vacuum cleaner':2,'Electric space heater':2,'Electric shower heater':2,'Fan':2,'Fridge':2,'Freezer':2}

FHMM=FHMM() #variable del modelo FHMM.train(train_Devices) #entrenamiento del modelo FHMM FHMM

And this is the error: Training model for submeter 'ElecMeter(instance=2, building=1, dataset='DSUAL', appliances=[Appliance(type='electric furnace', instance=1)])' with 3 states Training model for submeter 'ElecMeter(instance=3, building=1, dataset='DSUAL', appliances=[Appliance(type='microwave', instance=1)])' with 3 states Training model for submeter 'ElecMeter(instance=4, building=1, dataset='DSUAL', appliances=[Appliance(type='television', instance=1)])' with 3 states Training model for submeter 'ElecMeter(instance=5, building=1, dataset='DSUAL', appliances=[Appliance(type='kettle', instance=1)])' with 3 states Training model for submeter 'ElecMeter(instance=6, building=1, dataset='DSUAL', appliances=[Appliance(type='vacuum cleaner', instance=1)])' with 3 states Training model for submeter 'ElecMeter(instance=7, building=1, dataset='DSUAL', appliances=[Appliance(type='electric space heater', instance=1)])' with 3 states Training model for submeter 'ElecMeter(instance=8, building=1, dataset='DSUAL', appliances=[Appliance(type='electric shower heater', instance=1)])' with 3 states Training model for submeter 'ElecMeter(instance=9, building=1, dataset='DSUAL', appliances=[Appliance(type='fan', instance=1)])' with 3 states Training model for submeter 'ElecMeter(instance=10, building=1, dataset='DSUAL', appliances=[Appliance(type='fridge', instance=1)])' with 3 states Training model for submeter 'ElecMeter(instance=11, building=1, dataset='DSUAL', appliances=[Appliance(type='freezer', instance=1)])' with 3 states

---

MemoryError Traceback (most recent call last) Cell In[6], line 5 2 num_states_dict ={'Electric furnace':2, 'Microwave':2,'Television':2,'Kettle':2,'Vacuum cleaner':2,'Electric space heater':2,'Electric shower heater':2,'Fan':2,'Fridge':2,'Freezer':2} 4 FHMM=FHMM() #variable del modelo ----> 5 FHMM.train(train_Devices) #entrenamiento del modelo FHMM 6 FHMM

File ~/.local/lib/python3.8/site-packages/nilmtk/legacy/disaggregate/fhmm_exact.py:364, in FHMM.train(self, metergroup, num_states_dict, **load_kwargs) 361 # UGLY! But works. 362 self.meters.append(meter) --> 364 learnt_model_combined = create_combined_hmm(new_learnt_models) 365 self.individual = new_learnt_models 366 self.model = learnt_model_combined

File ~/.local/lib/python3.8/site-packages/nilmtk/legacy/disaggregate/fhmm_exact.py:116, in create_combined_hmm(model) 112 listmeans = [model[appliance].means.flatten().tolist() 113 for appliance in model] 115 pi_combined = compute_pi_fhmm(list_pi) --> 116 A_combined = compute_A_fhmm(list_A) 117 [mean_combined, cov_combined] = compute_means_fhmm(list_means) 119 combined_model = hmm.GaussianHMM(n_components=len(pi_combined), covariance_type='full')

File ~/.local/lib/python3.8/site-packages/nilmtk/legacy/disaggregate/fhmm_exact.py:75, in compute_A_fhmm(list_A) 73 result = list_A[0] 74 for i in range(len(list_A) - 1): ---> 75 result = np.kron(result, list_A[i + 1]) 76 return result

File <__array_function__ internals>:5, in kron(*args, **kwargs)

File ~/.local/lib/python3.8/site-packages/numpy/lib/shapebase.py:1154, in kron(a, b) 1152 bs = (1,)*(nda-ndb) + bs 1153 nd = nda -> 1154 result = outer(a, b).reshape(as+bs) 1155 axis = nd-1 1156 for _ in range(nd):

File <__array_function__ internals>:5, in outer(*args, **kwargs)

File ~/.local/lib/python3.8/site-packages/numpy/core/numeric.py:909, in outer(a, b, out) 907 a = asarray(a) 908 b = asarray(b) --> 909 return multiply(a.ravel()[:, newaxis], b.ravel()[newaxis, :], out)

MemoryError: Unable to allocate 26.0 GiB for an array with shape (387420489, 9) and data type float64

[annis-souames]

Do you have 26 GB of memory ? Even with 32 GB of RAM this might not work if you have other programs running. The solution would be either : Get more RAM (try training on the cloud) or reduce the size of your data by downsampling to 1 minutes, 5 minutes etc.

[crn565]

Thank you very much for your help. Unfortunally I haven't any machine with more than 32GB of RAM, so I decided to try to run nilmtk in google collab , but at the moment I haven't got any lucky. If I get to work nilmtk in collab , I'll share the code for the people that are interested in.

[Raphael164]

Thank you very much for your help. Unfortunally I haven't any machine with more than 32GB of RAM, so I decided to try to run nilmtk in google collab , but at the moment I haven't got any lucky. If I get to work nilmtk in collab , I'll share the code for the people that are interested in.

Did you manage to run NILMTK (and NILMTK-Contrib) in Google Collab?

[RajatPatodi]

Thank you very much for your help. Unfortunally I haven't any machine with more than 32GB of RAM, so I decided to try to run nilmtk in google collab , but at the moment I haven't got any lucky. If I get to work nilmtk in collab , I'll share the code for the people that are interested in.

Good evening @crn565 , @Raphael164 , I too am getting the same error even after using a system with 32 GB ram How did you resolved the issue?

[crn565]

I really probed executing fhmm with one Intel xeon with 64gb of Ram and the error is the same... I think the error is caused of a possible Bugs in the implementations of fhmm in Nilmtk. To fix the error you can reduce training time or reduce sampling time too