search

janclemenslab / das

Deep Audio Segmenter
http://janclemenslab.org/das/
27 stars 10 forks source link

Failure during testing phase of training #60

Closed chitayata closed 8 months ago

chitayata commented 1 year ago

Dear Jan,

While training a new network in the GUI the process fails during the testing phase and I get the following error. The model seems to be successfully created, but without the testing phase and the subsequent results.h5 I can't see how well it it is performing. I've tried twice and keep getting the same error because of a memory issue. Is there a way to just run the testing phase with the model that was produced to get around doing the entire training again?

Thanks! Adrienne

INFO:das.train:TESTING: INFO:das.train: Re-loading last best model from C:/Users/chita/OneDrive/Desktop/Chimp_vocalizations/Dataset_ChimpNetwork/Dataset3/Dataset3_1_99_701515.res/BaseRec_16_20230321_000240_model.h5. INFO:das.train: Predicting. 100%|████████████████████████████████████████████████████████████████████████████| 13386/13386 [07:00<00:00, 31.81it/s] INFO:das.train: Evaluating. Traceback (most recent call last): File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\xarray_behave\gui\utils.py", line 219, in run self.fn(*self.args, **self.kwargs) File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\das\train.py", line 482, in train conf_mat, report = evaluate.evaluate_segments(labels_test, labels_pred, params["class_names"], report_as_dict=True) File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\das\evaluate.py", line 34, in evaluate_segments conf_mat = sklearn.metrics.confusion_matrix(labels_test, labels_pred) File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\sklearn\metrics\_classification.py", line 307, in confusion_matrix y_type, y_true, y_pred = _check_targets(y_true, y_pred) File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\sklearn\metrics\_classification.py", line 111, in _check_targets unique_values = np.union1d(y_true, y_pred) File "<__array_function__ internals>", line 180, in union1d File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\numpy\lib\arraysetops.py", line 781, in union1d return unique(np.concatenate((ar1, ar2), axis=None)) File "<__array_function__ internals>", line 180, in unique File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\numpy\lib\arraysetops.py", line 274, in unique ret = _unique1d(ar, return_index, return_inverse, return_counts, File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\numpy\lib\arraysetops.py", line 328, in _unique1d ar = np.asanyarray(ar).flatten() numpy.core._exceptions._ArrayMemoryError: Unable to allocate 5.92 GiB for an array with shape (795021312,) and data type int64

postpop commented 1 year ago

Hi Adrienne,

unfortunately, it is currently not possible to only run the testing through the GUI. Only solution would be to reduce the size of the test data.

We have a notebook that tests a network trained on bird calls: https://janclemenslab.org/das/tutorials/evaluate_bird.html

I'm happy to help you get it to work with your data.

postpop commented 1 year ago

Hi Adrienne,

I just realized that we have functionality for testing a model using the command line. I've cleaned the code up and it should now work. To use the functionality, please update DAS to v0.30.0 from the command line: pip install das --upgrade --no-deps --force.

You can use it by calling das evaluate MODELSAVENAME, where MODELSAVENAME is the date-time trunk of the model file that was generated during fitting. It will load the dataset that was used for fitting the model and evaluate the model on the test data stored in the dataset.

Let me know if you run into problems.

chitayata commented 1 year ago

Great news, thank you!

I was going to post today that I tried reducing the testing set a couple times and still ran into trouble.

Will try this and let you know how it works out.

Much appreciated!

Best, Adrienne

On Sat, Mar 25, 2023, 9:27 AM Jan Clemens @.***> wrote:

Hi Adrienne,

I just realized that we have functionality for testing a model using the command line. I've cleaned the code up and it should now work. To use the functionality, please update DAS to v0.30.0 from the command line: pip install das --upgrade --no-deps --force.

You can use it by calling das evaluate MODELSAVENAME, where MODELSAVENAME is the date-time trunk of the model file that was generated during fitting. It will load the dataset that was used for fitting the model and evaluate the model on the test data stored in the dataset.

Let me know if you run into problems.

— Reply to this email directly, view it on GitHub https://github.com/janclemenslab/das/issues/60#issuecomment-1483837462, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2TIYLUAMFFKNFFJ42QUPJTW536D7ANCNFSM6AAAAAAWCOHFSQ . You are receiving this because you authored the thread.Message ID: @.***>