Small code changes for windows support.

[awsjpleger]

Dear Zhi, Thanks for sharing this awesome project. I would like to propose a few small code changes aiming at improving compatibility for windows operating system :

nuclei.io/setup.py - Replace :

with open("README.md") as readme_file:

with :

with open("README.md", encoding="utf8") as readme_file:

This fixes an installation issue causing the following error :

UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 1053: character maps to <undefined>

nuclei.io/feature_pre-calculation/nuc_stat.py and nuclei.io/feature_pre-calculation/nuc_stat_DL.py:

Make a small alteration to the parmap() function so that it detects if the operating system is windows based and falls back to using thread instead of process:

def parmap(f, X, nprocs=mp.cpu_count()):
    import platform
    q_in = mp.Queue(1)
    q_out = mp.Queue()
    if platform.system() == "Windows":
        import threading
        proc = [ threading.Thread(target=parfun, args=(f, q_in, q_out)) for _ in range(nprocs) ]
    else:
        proc = [mp.Process(target=parfun, args=(f, q_in, q_out)) for _ in range(nprocs)]

    for p in proc:
        p.daemon = True
        p.start()
    sent = [q_in.put((i, x)) for i, x in enumerate(X)]
    [q_in.put((None, None)) for _ in range(nprocs)]
    res = [q_out.get() for _ in range(len(sent))]
    [p.join() for p in proc]
    return [x for i, x in sorted(res)]

This fixes and issue caused by impossible pickling. ( There could potentially be a performance impact in switching from process to thread usage. )

Thanks,

JP.

[huangzhii]

Thank you! Feel free to make a pull request so the community will recognize your contribution to this repo! :)

[awsjpleger]

Hi Zhii. Thanks for you reply. I have tried to create a branch on this repo but got an Access Denied error. Is the repo access restricted at the moment by any chance? Happy to follow up by email or meeting if needed ( jpleger@amazon.com) .

Thanks.

JP.

[huangzhii]

Thank you for your contribution. PR has merged.