should I change the value of alpha when using my own data?

[marvin521]

Hi Sunil, Thank you for contributing to the wonderful code, I want to apply the method to my own data. Should I change the value of the alpha when performing the method? It looks like that the method cannot abstain samples on my own data using default parameter setting (including alpha).

[thulas]

Hi,

There are couple of things to try to increase abstention:

1. increase alpha_init_factor; try setting it to 128 ( default is 64).
2. Reduce the warm up period (learn_epochs parameter). Default is 10 epochs, try 5 epochs. Hope this helps.

• Sunil.

[hussam789]

I'm having the opposite issue, all of the training images are abstained, what alpha_init_factor and alpha_final you suggest i should use (on larger datasets 1k+ classes)?

[thulas]

How many epochs are you training for? Theoretically, if you train for long enough the abstention rate will go to zero. Anyway, you could try experimenting with a longer warmup period. Try setting learn_epochs to 20, 30 or higher to see if that helps.

[pingqingsheng]

I'm trying to running abstention on cifar10 with noisy level 0.2 using command

python train_dac.py --dataset cifar10 --datadir ../data --nesterov --net_type resnet2 --depth 34 -use-gpu --epochs 180 -cuda_device 3 --seed 0 --rand_labels 0.2 --loss_fn dac_loss

but I end up with the network abstained all my data points. How may get similar results as the paper suggested ? Are there other parameter to tune ?

Thank you so much !

Songzhu

[thulas]

@pingqingsheng: do you mean it abstains on all the training data or validation data, or both?

In any case, you could try reducing the alpha_init_factor from 64 (default) to 32 or 16. This makes initial abstention less aggressive.

[pingqingsheng]

Both.  I reduce it to 2, but I still don't get the ideal number.  It gives 82% on cifar10 with 0.2 noisy rate.  May I know the parameter setting for what you use in the paper?  What is the warm up epochs? Thank you so much!Sent from my Huawei phone-------- Original message --------From: Sunil Thulasidasan notifications@github.comDate: Thu, Sep 12, 2019, 4:36 PMTo: thulas/dac-label-noise dac-label-noise@noreply.github.comCc: pingqingsheng imzszhahahaha@gmail.com, Mention mention@noreply.github.comSubject: Re: [thulas/dac-label-noise] should I change the value of alpha when using my own data? (#1)@pingqingsheng: do you mean it abstains on all the training data or validation data, or both? In any case, you could try reducing the alpha_init_factor from 64 (default) to 32 or 16. This makes initial abstention less aggressive.

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

[thulas]

@pingqingsheng

Try the following:

Step 1: python train_dac.py --dataset cifar10 --nesterov --net_type resnet --depth 34 -use-gpu --epochs 165 --loss_fn dac_loss --learn_epochs 20 --rand_labels 0.2

Step 2: Remove the points from the training set that were abstained at the end of training and retrain with regular cross-entropy.

(PS: make sure you save the corrupted indices in step 1 and pass these onto step 2. see --help on how to do this. Otherwise, step 2 will have only clean labels)

[pingqingsheng]

Got it. Does the code itself perform the abstained points removing, or we need to do it ourselves?

On Thu, Sep 12, 2019 at 9:39 PM Sunil Thulasidasan notifications@github.com wrote:

@pingqingsheng https://github.com/pingqingsheng

Try the following: python train_dac.py --dataset cifar10 --nesterov --net_type resnet --depth 34 -use-gpu --epochs 165 --loss_fn dac_loss --learn_epochs 20 --rand_labels 0.2

Then remove the points from the training set that were abstained at the end of training and retrain with regular cross-entropy.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thulas/dac-label-noise/issues/1?email_source=notifications&email_token=AFRHC3APEEDR7OIV54TU7WDQJLVOPA5CNFSM4IKQIJG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6TXDII#issuecomment-531067297, or mute the thread https://github.com/notifications/unsubscribe-auth/AFRHC3HM4LQ6DJI2SJDBPI3QJLVOPANCNFSM4IKQIJGQ .

[thulas]

Got it. Does the code itself perform the abstained points removing, or we need to do it ourselves? …

Currently it does not remove or save a list of abstained samples; this is a feature I plan to add. For now, you can pass the --save_train_scores option which will save the predictions on the training set at every epoch, using which you can filter out the abstained samples.

[pingqingsheng]

I see. Tha't why I don't get the ideal score. So what is the testing score it gives me finally ?

On Fri, Sep 20, 2019 at 12:50 PM Sunil Thulasidasan < notifications@github.com> wrote:

Got it. Does the code itself perform the abstained points removing, or we need to do it ourselves? … <#m451320541917796016>

Currently it does not remove or save a list of abstained samples; this is a feature I plan to add. For now, you can pass the --save_train_scores option which will save the predictions on the training set at every epoch, using which you can filter out the abstained samples.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thulas/dac-label-noise/issues/1?email_source=notifications&email_token=AFRHC3GDGOSBYPMX2XZ736LQKT5PHA5CNFSM4IKQIJG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7HI27Q#issuecomment-533630334, or mute the thread https://github.com/notifications/unsubscribe-auth/AFRHC3HWHDQOOZYGJVRI5OLQKT5PHANCNFSM4IKQIJGQ .

[thulas]

You mean if you retrain on the cleaner (i.e non abstained set)? For CIFAR-10 with ResNet 34 and initially 20% label noise, after retraining you should get about 93.35% on the test set (see Table I from the ICML paper).

[pingqingsheng]

I mean in the original code, there is a phase-4 and it will give me a testing accuracy. What is that accuracy ?

On Fri, Sep 20, 2019 at 3:22 PM Sunil Thulasidasan notifications@github.com wrote:

You mean if you retrain on the cleaner (i.e non abstained set)? For CIFAR-10, with initially 20% label noise, and then after retraining, you should get about 93.35% on the test set.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thulas/dac-label-noise/issues/1?email_source=notifications&email_token=AFRHC3GOZ4BHPYQHK4B7UHDQKUPGFA5CNFSM4IKQIJG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7HU23I#issuecomment-533679469, or mute the thread https://github.com/notifications/unsubscribe-auth/AFRHC3FT7BL5TKDTTH63SYDQKUPGFANCNFSM4IKQIJGQ .

[thulas]

Step 1 Phase 4 accuracy will be low (and should be ignored) since many samples are abstained on at the end of training, and abstention label is not in the ground truth. It will probably be between 75 and 80%.

It is Step-2, Phase 4 accuracy -- i.e accuracy after removing noisy points -- that we report in the paper.

[pingqingsheng]

Got you.  For step two I don't train with abstained loss right?  Sent from my Huawei phone-------- Original message --------From: Sunil Thulasidasan notifications@github.comDate: Fri, Sep 20, 2019, 3:33 PMTo: thulas/dac-label-noise dac-label-noise@noreply.github.comCc: pingqingsheng imzszhahahaha@gmail.com, Mention mention@noreply.github.comSubject: Re: [thulas/dac-label-noise] should I change the value of alpha when using my own data? (#1)Step 1 Phase 4 accuracy will be low (and should be ignored) since many samples are abstained on at the end of training, and abstention label is not in the ground truth. It will probably be between 75 and 80%. It is Step-2, Phase 4 accuracy -- i.e accuracy after removing noisy points -- that we report in the paper.

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

[thulas]

Yes. Re-train using the regular cross-entropy loss.

On Sep 20, 2019, at 1:59 PM, pingqingsheng notifications@github.com wrote:

Got you. For step two I don't train with abstained loss right? Sent from my Huawei phone-------- Original message --------From: Sunil Thulasidasan notifications@github.comDate: Fri, Sep 20, 2019, 3:33 PMTo: thulas/dac-label-noise dac-label-noise@noreply.github.comCc: pingqingsheng imzszhahahaha@gmail.com, Mention mention@noreply.github.comSubject: Re: [thulas/dac-label-noise] should I change the value of alpha when using my own data? (#1)Step 1 Phase 4 accuracy will be low (and should be ignored) since many samples are abstained on at the end of training, and abstention label is not in the ground truth. It will probably be between 75 and 80%. It is Step-2, Phase 4 accuracy -- i.e accuracy after removing noisy points -- that we report in the paper.

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/thulas/dac-label-noise/issues/1?email_source=notifications&email_token=AAVHPAJ6G3EV5URI3MXIYWTQKUTSLA5CNFSM4IKQIJG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7HXOOQ#issuecomment-533690170, or mute the thread https://github.com/notifications/unsubscribe-auth/AAVHPAMAUDZP5ZHVAUFKXJTQKUTSLANCNFSM4IKQIJGQ.