search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
132.92k stars 26.52k forks source link

wandb metric argument is weird #24409

Closed edmcman closed 3 months ago

edmcman commented 1 year ago

System Info

Who can help?

@AyushExel

Information

Tasks

Reproduction

There are a few bugs/weirdness when specifying the metric in wandb hyperparameter search.

  1. If a metric is specified via the hp_space argument but there is not a metric argument to hyperparameter_search, the metric in the hp_space is ignored and changed to eval/loss. See: https://github.com/huggingface/transformers/blame/6ce6d62b6f20040129ec9831e7c4f6576402ea42/src/transformers/integrations.py#L497
  2. If a custom hp_space is provided that does not define metric at all, and the metric argument is specified to hyperparameter_search, this code throws an exception: https://github.com/huggingface/transformers/blame/6ce6d62b6f20040129ec9831e7c4f6576402ea42/src/transformers/integrations.py#L500

Expected behavior

  1. If a hp_space defines a metric, use that instead of overwriting it.
  2. Don't throw an exception is the hp_space lacks a metric key.
AyushExel commented 1 year ago

Thanks @edmcman for the ping.

@ayulockin @morganmcg1 in case this is relevant to the integrations team. tbh, I don't remember much about this integration but I implemented it so happy to help in case there's a blocker.

ayulockin commented 1 year ago

Thanks for the detailed issue @edmcman; we are taking a look at this. Will connect with you @AyushExel in case of blocker.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

edmcman commented 1 year ago

Ping

On Mon, Jul 31, 2023, 11:03 AM github-actions[bot] - @. <github.edmcman.99c9f1b9d0.notifications# @.> wrote:

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md are likely to be ignored.

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/24409#issuecomment-1658554442, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHYKZPH3FQDLIUC7DSYYRLXS7CKXANCNFSM6AAAAAAZPGOQMA . You are receiving this because you were mentioned.Message ID: @.***>

edmcman commented 1 year ago

Ping

On Fri, Aug 25, 2023, 4:03 AM github-actions[bot] - @. @.> wrote:

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md are likely to be ignored.

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/24409#issuecomment-1692940561, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHYKZPWTBMM362D7HGYQGTXXBL3JANCNFSM6AAAAAAZPGOQMA . You are receiving this because you were mentioned.Message ID: @.***>

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

edmcman commented 1 year ago

Ping

On Tue, Sep 19, 2023, 4:03 AM github-actions[bot] - @. @.> wrote:

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md are likely to be ignored.

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/24409#issuecomment-1725018773, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHYKZKLSBIWBWDK3UV6WGDX3FGVJANCNFSM6AAAAAAZPGOQMA . You are receiving this because you were mentioned.Message ID: @.***>

ayulockin commented 1 year ago

Hey @edmcman, apologies for the delay. I am taking a look at this. Will let you know soon.

edmcman commented 1 year ago

No worries, I was just keeping the issue open :)

edmcman commented 11 months ago

Boop

On Sat, Oct 14, 2023, 4:06 AM github-actions[bot] - @. @.> wrote:

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md are likely to be ignored.

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/24409#issuecomment-1762708379, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHYKZNRWB7DQT2CAGH3BZ3X7JBZVANCNFSM6AAAAAAZPGOQMA . You are receiving this because you were mentioned.Message ID: @.***>

edmcman commented 10 months ago

Boop

On Fri, Nov 10, 2023, 3:07 AM github-actions[bot] - @. @.> wrote:

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md are likely to be ignored.

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/24409#issuecomment-1805270715, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHYKZLZLJX6TNBZJ2LOJG3YDXOFZAVCNFSM6AAAAAAZPGOQMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBVGI3TANZRGU . You are receiving this because you were mentioned.Message ID: @.***>

github-actions[bot] commented 10 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

edmcman commented 10 months ago

Boop

On Tue, Dec 5, 2023, 3:06 AM github-actions[bot] - @. @.> wrote:

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md are likely to be ignored.

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/24409#issuecomment-1840213033, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHYKZJANX7W6GOKOVQQOWLYH3IZZAVCNFSM6AAAAAAZPGOQMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBQGIYTGMBTGM . You are receiving this because you were mentioned.Message ID: @.***>

github-actions[bot] commented 9 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

edmcman commented 9 months ago

Boop

On Sat, Dec 30, 2023, 3:06 AM github-actions[bot] - @. @.> wrote:

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md are likely to be ignored.

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/24409#issuecomment-1872479380, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHYKZMEV7QFVIKKLL7VA3TYL7DP7AVCNFSM6AAAAAAZPGOQMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGQ3TSMZYGA . You are receiving this because you were mentioned.Message ID: @.***>

github-actions[bot] commented 8 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

edmcman commented 8 months ago

Bump

On Wed, Jan 24, 2024, 3:07 AM github-actions[bot] - @. @.> wrote:

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md are likely to be ignored.

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/24409#issuecomment-1907600067, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHYKZJIJOVQKD7V5UOB2GDYQC6NPAVCNFSM6AAAAAAZPGOQMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBXGYYDAMBWG4 . You are receiving this because you were mentioned.Message ID: @.***>

github-actions[bot] commented 7 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

edmcman commented 7 months ago

Boop

On Sun, Feb 18, 2024, 3:07 AM github-actions[bot] - @. @.> wrote:

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md are likely to be ignored.

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/24409#issuecomment-1950997682, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHYKZKVITBHJR65I2F5KTLYUGZCFAVCNFSM6AAAAAAZPGOQMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJQHE4TONRYGI . You are receiving this because you were mentioned.Message ID: @.***>

github-actions[bot] commented 6 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

amyeroberts commented 6 months ago

Hi @AyushExel - any update on this?

AyushExel commented 6 months ago

@amyeroberts hey, I'm no longer associated with W&B. @ayulockin should be able to help. Thanks!

edmcman commented 5 months ago

@ayulockin can you help?

On Mon, Apr 8, 2024, 4:09 AM github-actions[bot] - @. @.> wrote:

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md are likely to be ignored.

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/24409#issuecomment-2042121322, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHYKZIUIH7S7OX4CA2SP6DY4JGD5AVCNFSM6AAAAAAZPGOQMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBSGEZDCMZSGI . You are receiving this because you were mentioned.Message ID: @.***>

edmcman commented 5 months ago

.

On Fri, May 3, 2024, 4:09 AM github-actions[bot] - @. @.> wrote:

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md are likely to be ignored.

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/24409#issuecomment-2092520316, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHYKZK63QRSGSNWJ6OHLDTZANA2JAVCNFSM6AAAAAAZPGOQMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJSGUZDAMZRGY . You are receiving this because you were mentioned.Message ID: @.***>

amyeroberts commented 5 months ago

cc @parambharat as they've been doing loads of great integration work between HF & W&B recently

github-actions[bot] commented 4 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.