17zuoye / pyirt

A python library of IRT algorithm
MIT License
99 stars 54 forks source link

AttributeError: Can't pickle local object 'IRT_MMLE_2PL.__update_theta_distr.<locals>.update' #22

Open nix1 opened 3 years ago

nix1 commented 3 years ago

I'm trying to use PyIRT, however it seems that the library has some issues with how it uses multiprocessing. I guess it used to be fine some time ago, I'm currently using Python 3.9.1.

It seems that essentially PyIRT passes a local update function as a target to Process. In effect, when multiprocessing tries to fork and spawn a new process, it tries to dump a pickle of the function, which fails with AttributeError: Can't pickle local object 'IRT_MMLE_2PL.__update_theta_distr.<locals>.update'.

Does anyone have a solution or a workaround to this issue?

nix1 commented 3 years ago

For now, I just removed creating processes in pyirt/solver/model.py. It probably became slower, but it worked.

So for instance:

        for i in range(num_chunk):
            p = mp.Process(target=update, args=(user_ell, user_cnt, chunk_list[i][0], chunk_list[i][1],))
            procs.append(p)
        if num_chunk > 1:
            procs = procs_operator(procs, 1200, self.check_interval)
        else:
            procs = procs_operator(procs, 7200, 0.1)

becomes:

        for i in range(num_chunk):
            update(user_ell, user_cnt, chunk_list[i][0], chunk_list[i][1])

A similar situation happens around 4 times in this file.