[BUG] `parallelize` function with python modules results to `seg fault`

toiletsandpaper commented 7 months ago

Bug description

I'm trying to parallelize my classify function over a DynamicVector of custom structs. But without parallelization code is very slow, even slower than python's approach, but this is probably because of my skill issue with sorting.

Firstly, I've got a problem that without try-catching function that raises - I can't use any of parallelize function at all. Secondly, when I wrote somewhat a workaround - I got very short seg fault

Please submit a bug report to https://github.com/modularml/mojo/issues and include the crash backtrace along with all the relevant source codes.
[5581:16081077:20240318,154326.598003:WARNING crash_report_exception_handler.cc:257] UniversalExceptionRaise: (os/kern) failure (5)
zsh: segmentation fault  mojo run main.mojo

Steps to reproduce

error raises here - https://github.com/toiletsandpaper/mojo_zlib_classification/blob/f463f042bee8287da1b646cabe19460872831e75/classifiers/classifier.mojo#L42C1-L60C23
Just follow a readme of seg_fault_parallelized branch and it should install all with make locally https://github.com/toiletsandpaper/mojo_zlib_classification/blob/seg_fault_parallelized/README.md

System information

❯ uname -moprsv
Darwin 23.3.0 Darwin Kernel Version 23.3.0: Wed Dec 20 21:30:27 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T8103 arm64 arm
❯ modular -v
modular 0.5.2 (6b3a04fd)
❯ mojo -v
mojo 24.1.0 (55ec12d6)

  Host Information
  ================

  Target Triple: aarch64-unknown-macosx
  CPU: apple-m1
  CPU Features: aes, complxnum, crc, dotprod, fp-armv8, fp16fml, fullfp16, jsconv, lse, neon, pauth, ras, rcpc, rdm, sha2, sha3

toiletsandpaper commented 7 months ago

Looks like its caused by parallelization of python import. Minimal repro:

from python import Python
from algorithm import parallelize

fn do_something_python(text: String) raises -> PythonObject:
    var uuid = Python.import_module('uuid')
    return text + '\t' + str(uuid.uuid4())

fn main() raises:
    var texts = DynamicVector[String]()
    texts.append("Hello, World!")
    texts.append("This is a test.")
    texts.append("Hello, World!")
    texts.append("This is a test.")
    texts.append("Hello, World!")
    texts.append("This is a test.")
    texts.append("Hello, World!")
    texts.append("This is a test.")
    texts.append("Hello, World!")
    texts.append("This is a test.")

    # works just right
    # for i in range(len(texts)):
    #     print(do_something_python(texts[i]))

    @parameter
    fn _do(i: Int):
        try:
            print(do_something_python(texts[i]))
        except:
            print('Error')

    # parallelize[_do](1)  # returns only UUID4, but not text... why?
    parallelize[_do](len(texts))

p.s. IDK why, but paralellize[_do](1) outputs something, that only Scooby Doo team is able to investigate

toiletsandpaper commented 7 months ago

By the way, ffi.DLHandle -> get_function also doesn't work, resulting the same error

modularml / mojo