search

snwagh / falcon-public

Implementation of protocols in Falcon
90 stars 46 forks source link

New bug - Segmentation fault on layer push_back on VGG16 #34

Open HuangPZ opened 2 years ago

HuangPZ commented 2 years ago 
network = "VGG16";
dataset = "CIFAR10"; // or ImageNet

Segmentation fault is detected when pushing back the 31st layer (an FC layer) to layers. This occurred on my two machines so I think it's a machine issue. Tried to fix it myself but didn't quite get the reason for the issue. I see the repo is updated a lot since I first checked, please do investigate this issue to see if the bug can be reproduced.

snwagh commented 2 years ago

There haven't been any major changes to that part of the code. Can you please paste the changes you've made and the machine config you're running it on? It is nearly impossible to debug without a concrete bug. An issue occurring on two out of three machines would indicate that it might not be an issue with the repo.

HuangPZ commented 2 years ago

Hi! I meant running make terminal using localhost setting on a single machine, and I encountered the error. I tried on two different machines and got the same error. I've updated to the newest version of the code when commenting.

snwagh commented 2 years ago

Thanks for spotting the bug, I was able to reproduce it on some machine.

The issue is arising from the constructor of the FC Layer class and more specifically from the initialization (these few lines of code). I still do not know what the cause of the seg fault is but maybe you can take a stab at debugging it from here?

@AndesPooh258 Since you worked on this commit (3d3e8f2c549df16118d7962137266f144158f2b4), can you please help @HuangPZ with this bug?

andeskyl commented 2 years ago

As I am dealing with mutliple project deadlines, I am sorry that I may not have time to help at this moment. But the high level idea of this modified part of code is to initialize non-zero weights for the FC layer. That said, if you don't mind to initialize the weight of FC layer as 0, you may feel free to comment these part of code and see if this error occurs.

HuangPZ commented 2 years ago

As I am dealing with mutliple project deadlines, I am sorry that I may not have time to help at this moment. But the high level idea of this modified part of code is to initialize non-zero weights for the FC layer. That said, if you don't mind to initialize the weight of FC layer as 0, you may feel free to comment these part of code and see if this error occurs.

This indeed worked. Maybe we want to figure out what's happening later.