neurodata / df-dn-paper

Conceptual & empirical comparisons between decision forests & deep networks
https://dfdn.neurodata.io
Other
17 stars 8 forks source link

BUG Error in combining train & valid loader for Vision #65

Open adwaykanhere opened 2 years ago

adwaykanhere commented 2 years ago

When creating train+val loaders for tuning CNNs for vision data, the following traceback is received:

INFO 05-12 21:06:37] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter lr. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 05-12 21:06:37] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter momentum. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 05-12 21:06:37] ax.service.utils.instantiation: Inferred value type of ParameterType.INT for parameter epoch. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 05-12 21:06:37] ax.service.utils.instantiation: Inferred value type of ParameterType.STRING for parameter optimizer. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 05-12 21:06:37] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='lr', parameter_type=FLOAT, range=[1e-06, 0.4], log_scale=True), RangeParameter(name='momentum', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='epoch', parameter_type=INT, range=[15, 40]), ChoiceParameter(name='optimizer', parameter_type=STRING, values=['SGD', 'Adam'], is_ordered=False, sort_values=False)], parameter_constraints=[]).
[INFO 05-12 21:06:37] ax.modelbridge.dispatch_utils: Using Bayesian optimization since there are more ordered parameters than there are categories for the unordered categorical parameters.
[INFO 05-12 21:06:37] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 8 trials, GPEI for subsequent trials]). Iterations after 8 will take longer to generate due to  model-fitting.
[INFO 05-12 21:06:37] ax.service.managed_loop: Started full optimization with 20 steps.
[INFO 05-12 21:06:37] ax.service.managed_loop: Running optimization trial 1...
[INFO 05-12 21:06:39] ax.service.managed_loop: Running optimization trial 2...
[INFO 05-12 21:06:40] ax.service.managed_loop: Running optimization trial 3...
[INFO 05-12 21:06:41] ax.service.managed_loop: Running optimization trial 4...
[INFO 05-12 21:06:43] ax.service.managed_loop: Running optimization trial 5...
[INFO 05-12 21:06:44] ax.service.managed_loop: Running optimization trial 6...
[INFO 05-12 21:06:45] ax.service.managed_loop: Running optimization trial 7...
[INFO 05-12 21:06:47] ax.service.managed_loop: Running optimization trial 8...
[INFO 05-12 21:06:48] ax.service.managed_loop: Running optimization trial 9...
[INFO 05-12 21:06:51] ax.service.managed_loop: Running optimization trial 10...
[INFO 05-12 21:06:52] ax.service.managed_loop: Running optimization trial 11...
[INFO 05-12 21:06:54] ax.service.managed_loop: Running optimization trial 12...
[INFO 05-12 21:06:56] ax.service.managed_loop: Running optimization trial 13...
[INFO 05-12 21:06:58] ax.service.managed_loop: Running optimization trial 14...
[INFO 05-12 21:07:00] ax.service.managed_loop: Running optimization trial 15...
[INFO 05-12 21:07:02] ax.service.managed_loop: Running optimization trial 16...
[INFO 05-12 21:07:04] ax.service.managed_loop: Running optimization trial 17...
[INFO 05-12 21:07:06] ax.service.managed_loop: Running optimization trial 18...
[INFO 05-12 21:07:08] ax.service.managed_loop: Running optimization trial 19...
[INFO 05-12 21:07:10] ax.service.managed_loop: Running optimization trial 20...
[INFO 05-12 21:07:10] ax.modelbridge.base: Untransformed parameter 0.40000000000000013 greater than upper bound 0.4, clamping
[WARNING 05-12 21:07:12] ax.modelbridge.cross_validation: Metric accuracy was unable to be reliably fit.
[WARNING 05-12 21:07:12] ax.service.utils.best_point: Model fit is poor; falling back on raw data for best point.
[WARNING 05-12 21:07:12] ax.service.utils.best_point: Model fit is poor and data on objective metric accuracy is noisy; interpret best points results carefully.
(array([], dtype=int64),)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
[<ipython-input-15-8f0b5b112073>](https://localhost:8080/#) in <module>()
     44 )
     45 
---> 46 run_cnn32()

[<ipython-input-14-5a5512f0afa6>](https://localhost:8080/#) in run_cnn32()
    369             for i in train_valid_indices:
    370               print(np.where(classes_new == dataset_copy.targets[i]))
--> 371               dataset_copy.targets[i] = np.where(classes_new == dataset_copy.targets[i])[0][0]
    372 
    373             train_valid_sampler = torch.utils.data.sampler.SubsetRandomSampler(train_valid_indices)

IndexError: index 0 is out of bounds for axis 0 with size 0
PSSF23 commented 2 years ago

Was this commented out when you run create_train_loader?

  # for i in train_idxs:
  #    dataset.targets[i] = np.where(classes == dataset.targets[i])[0][0]
adwaykanhere commented 2 years ago

Yes, But that was because I had initially transformed train indices in create_loaders_final

PSSF23 commented 2 years ago

You need to use deep copies of the dataset on that, too. There's no point of deep copies if you don't save the original. Then the copy becomes "original."

adwaykanhere commented 2 years ago

Yes, I'm using a deepcopy of the dataset for the create_train_loader and create_loaders_final. I'm still getting the error.

PSSF23 commented 2 years ago

You have to create a deep copy each time you use those functions.

adwaykanhere commented 2 years ago

If I add deep copies before using each function, I get the error - IndexError: Target 7 is out of bounds.

PSSF23 commented 2 years ago

Create deep copies inside the functions

adwaykanhere commented 2 years ago

I'm still getting the same error even after creating copies inside the functions