euagendas / m3inference

A deep learning system for demographic inference (gender, age, and individual/person) that was trained on massive Twitter dataset using profile images, screen names, names, and biographies
http://www.euagendas.org
GNU Affero General Public License v3.0
145 stars 57 forks source link

Add support for win 64-bit #11

Open computermacgyver opened 3 years ago

computermacgyver commented 3 years ago

Per #10 , the multiprocessing in pytorch requires that

  1. The contents of my main program are within a
    if __name__ == "__main__":

    block

  2. The first statement of that block is freeze_support()
  3. The freeze_support function is imported from multiprocessing, i.e.,
    from multiprocessing import freeze_support

I've adapted scripts/m3twitter.py and scripts/preprocess.py to follow this convention.

I have confirmed these changes have no effect on linux (as intended), but do not have access to a 64-bit windows machine to test. (I have an old virtual box with a 32-bit install of windows but pytorch is not available for 32-bit systems)

zijwang commented 3 years ago

Thanks @computermacgyver for making the changes. From here, it looks like the code needs to be refactored into the following structure

import torch

def main()
    for i, data in enumerate(dataloader):
        # do something here

if __name__ == '__main__':
    main()

I unfortunately also don't have a windows machine to test things out. We could wait @anudeike to see whether the current changes work, and if not, we may try the recommended structure.

computermacgyver commented 3 years ago

Nice find on that documentation. I think that looks good and will do both that pattern and freeze_support. I wonder if I can spin up a Windows machine in the cloud---I'll see whats possible.