RelU activation at the end of generator

[MichalisLazarou]

Hey many thanks on your work, I was wondering whether using ReLU activation at the end of the generator works well in your impleenttion, i found that I had some issues training with the feature hallucinator with ReLU and I was wondering if you observed something similar

[KandariSrinivas]

Hi

Sorry for replying late. The paper says to use ReLu at the end of generator, however it is wrong. Because ReLU makes negative values zeros and target context vector does have negative values, so ReLU at the end will never produce context vector with good accuracy. I replaced it with sigmoid you can also try tanh

Thanks Srinivas

On Thu, Mar 11, 2021, 11:05 AM MichalisLazarou @.***> wrote:

Hey many thanks on your work, I was wondering whether using ReLU activation at the end of the generator works well in your impleenttion, i found that I had some issues training with the feature hallucinator with ReLU and I was wondering if you observed something similar

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/KandariSrinivas/Adversarial-Feature-Hallucination-Networks-for-Few-Shot-Learning/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL463OPHT525A3GKFXJ2RB3TDDS3DANCNFSM4ZARVZKQ .

[MichalisLazarou]

Hey Srinivas,

Many thanks for your reply. Indeed i also find that sigmoid works much better but i do mot understand why. Also sigmoid function makes the context vector positive. But if u see the featuree from the output of the resnet they are all positive and there are some that are larger than 1, which means that sigmoid cannot learn that because everything is mapped to 0 and 1. Any other intuition on why do you think sigmoid works better??

---

From: KandariSrinivas @.> Sent: Saturday, March 13, 2021 10:13 PM To: KandariSrinivas/Adversarial-Feature-Hallucination-Networks-for-Few-Shot-Learning @.> Cc: MichalisLazarou @.>; Author @.> Subject: Re: [KandariSrinivas/Adversarial-Feature-Hallucination-Networks-for-Few-Shot-Learning] RelU activation at the end of generator (#1)

Hi

Sorry for replying late. The paper says to use ReLu at the end of generator, however it is wrong. Because ReLU makes negative values zeros and target context vector does have negative values, so ReLU at the end will never produce context vector with good accuracy. I replaced it with sigmoid you can also try tanh

Thanks Srinivas

On Thu, Mar 11, 2021, 11:05 AM MichalisLazarou @.***> wrote:

Hey many thanks on your work, I was wondering whether using ReLU activation at the end of the generator works well in your impleenttion, i found that I had some issues training with the feature hallucinator with ReLU and I was wondering if you observed something similar

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/KandariSrinivas/Adversarial-Feature-Hallucination-Networks-for-Few-Shot-Learning/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL463OPHT525A3GKFXJ2RB3TDDS3DANCNFSM4ZARVZKQ .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FKandariSrinivas%2FAdversarial-Feature-Hallucination-Networks-for-Few-Shot-Learning%2Fissues%2F1%23issuecomment-798778458&data=04%7C01%7C%7Cf9c1c04dd1cd418903c808d8e65c713e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637512632005776075%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=lfaFYp2zMEbvli%2FOSg1t0XYiI%2BguxtWyYJSUyXFUy3g%3D&reserved=0, or unsubscribehttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAE34XYRYW26SQ2SG2SSN5R3TDPBN3ANCNFSM4ZARVZKQ&data=04%7C01%7C%7Cf9c1c04dd1cd418903c808d8e65c713e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637512632005776075%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=pXDYg0PFzQALtVsg6pWI6mxHxkioPSjS28C2%2Fp0vI0I%3D&reserved=0.