NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.29k stars 13.54k forks source link

`python3Packages.frozendict` is failing its tests on darwin #146765

Open risicle opened 2 years ago

risicle commented 2 years ago

Describe the bug

The tests for python3Packages.frozendict are currently failing on darwin and I don't think it's just a test issue and it may not be an upstream issue. Upstream cannot reproduce: https://github.com/Marco-Sulla/python-frozendict/issues/37 (ignore issue title, upstream renamed it)

Steps To Reproduce

Steps to reproduce the behavior:

  1. Be on darwin (tested x86_64, macos 10.15)
  2. Have a current nixpkgs
  3. Set doCheck = false in pkgs/development/python-modules/frozendict/default.nix
  4. Running the test suite is overkill to exhibit the issue, simply:
    $ NIX_PATH="nixpkgs=$(pwd)" nix-shell -p python38Packages.frozendict --pure
    [nix-shell:~/nixpkgs]$ python
    Python 3.8.12 (default, Nov  6 2021, 01:30:52) 
    [Clang 7.1.0 (tags/RELEASE_710/final)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import frozendict
    >>> frozendict.c_ext
    True
    >>> import pickle
    >>> pickle.dumps(frozendict.frozendict({1:2}))
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict

Additional context

A valid way forward could be to simply disable these tests on darwin, because I gather the python-only variant of this package exhibits this same behaviour and so many/most people who just do a naive pip install frozendict just get this behaviour anyway. Alternatively we could disable building the c extension entirely on darwin in case some downstream code gets confused by frozendict.c_ext being True yet pickling not working properly.

Notify maintainers

@NixOS/darwin-maintainers

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: python3Packages.frozendict
# a list of nixos modules affected by the problem
module:
starcraft66 commented 2 years ago

Set doCheck = false in pkgs/development/python-modules/frozendict/default.nix

We can simply add that test to disabledTests instead of disabling the test suite entirely. Would you like to open a PR?

risicle commented 2 years ago

That was only a step to enable the simple test to be reproduced.

We could indeed disable just the pickle tests, but as I noted it might lead to a less weird result to just turn off the c extension.

starcraft66 commented 2 years ago

How about adding an argument to the derivation for building the C extension? It should default to off, but a user could easily enable it at their own risk for the time being (at least until we figure out the underlying issue).

Marco-Sulla commented 2 years ago

Hello, I'm the new maintainer of frozendict.

I thought that @risicle simply tried to compile frozendict on MacOS. It seems on the contrary that it's using this package manager. Can you please print frozendict.__version__?

starcraft66 commented 2 years ago

@Marco-Sulla I tried upgrading the nix derivation for frozendict to the latest commit you released 5 hours ago and there seems to be a lot of tests failing due to pickle errors in our build environment.

Can you please print frozendict.__version__?

# If I disable tests
[nix-shell:~/src/nixpkgs]$ python
Python 3.9.6 (default, Oct 31 2021, 15:37:00) 
[Clang 7.1.0 (tags/RELEASE_710/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>> import frozendict
>> frozendict.__version__
'2.1.0'

Build log:

@NightmareMoon:~/s/nixpkgs (master) ❯ nix run 'nixpkgs#nix-update' python3Packages.frozendict
[...]
Update 2.0.6 -> 2.1.0 in /Users/tristan/src/nixpkgs/pkgs/development/python-modules/frozendict/default.nix
[...]
The package python3.9-frozendict-2.1.0 will be fetched as follows:
> import /Users/tristan/src/nixpkgs/pkgs/development/interpreters/python/fetchpypi.nix {
>   pname = "frozendict";
>   sha256 = "3f00de72805cf4c9e81b334f3f04809278b967d2fed84552313a0fcce511beb1";
>   version = "2.1.0";
> }
@NightmareMoon:~/s/nixpkgs (master) took 21s ❯ NIX_PATH=nixpkgs=. nix-shell --pure -p python3.pkgs.frozendict
this derivation will be built:
  /nix/store/7anksnsb8jnl4pi4nr5mkh7kr1aqv459-python3.9-frozendict-2.1.0.drv
[...]
Executing pytestCheckPhase                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
============================= test session starts ==============================                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
platform darwin -- Python 3.9.6, pytest-6.2.5, py-1.10.0, pluggy-1.0.0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
rootdir: /private/tmp/nix-build-python3.9-frozendict-2.1.0.drv-0/frozendict-2.1.0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
collected 209 items / 4 deselected / 205 selected                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

test/test_frozendict.py ................................................ [ 23%]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
....                                                                     [ 25%]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
test/test_frozendict_c.py ........................FFFFFF........FFF..... [ 47%]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
.......                                                                  [ 51%]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
test/test_frozendict_c_subclass.py .....................FFFFFF.......... [ 69%]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
.............                                                            [ 75%]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
test/test_frozendict_subclass.py ....................................... [ 94%]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
...........                                                              [100%]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 

=================================== FAILURES ===================================                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
________________________________ test_pickle[0] ________________________________                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                

fd = frozendict.frozendict({'Sulla': 'Marco', 'Hicks': 'Bill', frozendict.frozendict({1: 2}): 'frozen'})                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
protocol = 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    

>   ???                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         

<string>:124:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 

self = {'Sulla': 'Marco', 'Hicks': 'Bill', frozendict.frozendict({1: 2}): 'frozen'}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
proto = 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       

    def _reduce_ex(self, proto):                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
        assert proto < 2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
        cls = self.__class__                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        for base in cls.__mro__:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
            if hasattr(base, '__flags__') and not base.__flags__ & _HEAPTYPE:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
                break                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
            new = base.__new__                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
            if isinstance(new, _new_type) and new.__self__ is base:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
                break                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
        else:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
            base = object # not really reachable                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
        if base is object:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
            state = None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
        else:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
            if base is cls:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
>               raise TypeError(f"cannot pickle {cls.__name__!r} object")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
E               TypeError: cannot pickle 'dict' object                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          

/nix/store/lrw7ifwnpz7rskfkphq9mj9fwm5i9ldc-python3-3.9.6/lib/python3.9/copyreg.py:70: TypeError                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
________________________________ test_pickle[1] ________________________________                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                

fd = frozendict.frozendict({'Sulla': 'Marco', 'Hicks': 'Bill', frozendict.frozendict({1: 2}): 'frozen'})                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
protocol = 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    

>   ???                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         

<string>:124:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 

self = {'Sulla': 'Marco', 'Hicks': 'Bill', frozendict.frozendict({1: 2}): 'frozen'}                                                         
proto = 1                                                                                                                                   

    def _reduce_ex(self, proto):                                                                                                            
        assert proto < 2                                                                                                                    
        cls = self.__class__                                                                                                                
        for base in cls.__mro__:                                                                                                            
            if hasattr(base, '__flags__') and not base.__flags__ & _HEAPTYPE:                                                               
                break                                                                                                                       
            new = base.__new__                                                                                                              
            if isinstance(new, _new_type) and new.__self__ is base:                                                                         
                break                                                                                                                       
        else:                                                                                                                               
            base = object # not really reachable                                                                                            
        if base is object:                                                                                                                  
            state = None                                                                                                                    
        else:                                                                                                                               
            if base is cls:                                                                                                                 
>               raise TypeError(f"cannot pickle {cls.__name__!r} object")                                                                   
E               TypeError: cannot pickle 'dict' object                                                                                      

/nix/store/lrw7ifwnpz7rskfkphq9mj9fwm5i9ldc-python3-3.9.6/lib/python3.9/copyreg.py:70: TypeError                                            
________________________________ test_pickle[2] ________________________________                                                            

fd = frozendict.frozendict({'Sulla': 'Marco', 'Hicks': 'Bill', frozendict.frozendict({1: 2}): 'frozen'})                                    
protocol = 2                                                                                                                                

>   ???                                                                                                                                     
E   _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict                                           

<string>:124: PicklingError                                                                                                                 
________________________________ test_pickle[3] ________________________________                                                            

fd = frozendict.frozendict({'Sulla': 'Marco', 'Hicks': 'Bill', frozendict.frozendict({1: 2}): 'frozen'})                                    
protocol = 3                                                                                                                                

>   ???                                                                                                                                     
E   _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict                                           

<string>:124: PicklingError                                                                                                                 
________________________________ test_pickle[4] ________________________________                                                            

fd = frozendict.frozendict({'Sulla': 'Marco', 'Hicks': 'Bill', frozendict.frozendict({1: 2}): 'frozen'})                                    
protocol = 4                                                                                                                                

>   ???                                                                                                                                     
E   _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict                                           

<string>:124: PicklingError                                                                                                                 
________________________________ test_pickle[5] ________________________________                                                            

fd = frozendict.frozendict({'Sulla': 'Marco', 'Hicks': 'Bill', frozendict.frozendict({1: 2}): 'frozen'})                                    
protocol = 5                                                                                                                                

>   ???                                                                                                                                     
E   _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict                                           

<string>:124: PicklingError                                                                                                                 
__________________________________ test_repr ___________________________________                                                            

fd = frozendict.frozendict({'Sulla': 'Marco', 'Hicks': 'Bill', frozendict.frozendict({1: 2}): 'frozen'})                                    
fd_dict = {'Sulla': 'Marco', 'Hicks': 'Bill', frozendict.frozendict({1: 2}): 'frozen'}                                                      

>   ???                                                                                                                                     
E   AssertionError                                                                                                                          

<string>:158: AssertionError                                                                                                                
___________________________________ test_str ___________________________________                                                            

fd = frozendict.frozendict({'Sulla': 'Marco', 'Hicks': 'Bill', frozendict.frozendict({1: 2}): 'frozen'})                                    
fd_dict = {'Sulla': 'Marco', 'Hicks': 'Bill', frozendict.frozendict({1: 2}): 'frozen'}                                                      

>   ???                                                                                                                                     
E   AssertionError                                                                                                                          

<string>:162: AssertionError
_________________________________ test_format __________________________________                                                            

fd = frozendict.frozendict({'Sulla': 'Marco', 'Hicks': 'Bill', frozendict.frozendict({1: 2}): 'frozen'})                                    
fd_dict = {'Sulla': 'Marco', 'Hicks': 'Bill', frozendict.frozendict({1: 2}): 'frozen'}                                                      

>   ???                                                                                                                                     
E   AssertionError                                                                                                                          

<string>:166: AssertionError                                                                                                                
________________________________ test_pickle[0] ________________________________                                                            

fd = F({'Sulla': 'Marco', 'Hicks': 'Bill', F({1: 2}): 'frozen'}), protocol = 0                                                              

>   ???                                                                                                                                     

<string>:124:                                                                                                                               
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _                                                             

self = {'Sulla': 'Marco', 'Hicks': 'Bill', F({1: 2}): 'frozen'}, proto = 0                                                                  

    def _reduce_ex(self, proto):                                                                                                            
        assert proto < 2                                                                                                                    
        cls = self.__class__                                                                                                                
        for base in cls.__mro__:                                                                                                            
            if hasattr(base, '__flags__') and not base.__flags__ & _HEAPTYPE:                                                               
                break                                                                                                                       
            new = base.__new__                                                                                                              
            if isinstance(new, _new_type) and new.__self__ is base:                                                                         
                break                                                                                                                       
        else:                                                                                                                               
            base = object # not really reachable                                                                                            
        if base is object:                                                                                                                  
            state = None                                                                                                                    
        else:                                                                                                                               
            if base is cls:                                                                                                                 
>               raise TypeError(f"cannot pickle {cls.__name__!r} object")                                                                   
E               TypeError: cannot pickle 'dict' object                                                                                      

/nix/store/lrw7ifwnpz7rskfkphq9mj9fwm5i9ldc-python3-3.9.6/lib/python3.9/copyreg.py:70: TypeError                                            
________________________________ test_pickle[1] ________________________________                                                            

fd = F({'Sulla': 'Marco', 'Hicks': 'Bill', F({1: 2}): 'frozen'}), protocol = 1                                                              

>   ???                                                                                                                                     

<string>:124:                                                                                                                               
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _                                                             

self = {'Sulla': 'Marco', 'Hicks': 'Bill', F({1: 2}): 'frozen'}, proto = 1                                                                  

    def _reduce_ex(self, proto):                                                                                                            
        assert proto < 2                                                                                                                    
        cls = self.__class__                                                                                                                
        for base in cls.__mro__:                                                                                                            
            if hasattr(base, '__flags__') and not base.__flags__ & _HEAPTYPE:                                                               
                break                                                                                                                       
            new = base.__new__                                                                                                              
            if isinstance(new, _new_type) and new.__self__ is base:                                                                         
                break                                                                                                                       
        else:                                                                                                                               
            base = object # not really reachable                                                                                            
        if base is object:                                                                                                                  
            state = None                                                                                                                    
        else:                                                                                                                               
            if base is cls:                                                                                                                 
>               raise TypeError(f"cannot pickle {cls.__name__!r} object")                                                                   
E               TypeError: cannot pickle 'dict' object                                                                                      

/nix/store/lrw7ifwnpz7rskfkphq9mj9fwm5i9ldc-python3-3.9.6/lib/python3.9/copyreg.py:70: TypeError                                            
________________________________ test_pickle[2] ________________________________                                                            

fd = F({'Sulla': 'Marco', 'Hicks': 'Bill', F({1: 2}): 'frozen'}), protocol = 2                                                              

>   ???                                                                                                                                     
E   _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict                                           

<string>:124: PicklingError                                                                                                                 
________________________________ test_pickle[3] ________________________________                                                            

fd = F({'Sulla': 'Marco', 'Hicks': 'Bill', F({1: 2}): 'frozen'}), protocol = 3                                                              

>   ???                                                                                                                                     
E   _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict                                           

<string>:124: PicklingError                                                                                                                 
________________________________ test_pickle[4] ________________________________                                                            

fd = F({'Sulla': 'Marco', 'Hicks': 'Bill', F({1: 2}): 'frozen'}), protocol = 4                                                              

>   ???                                                                                                                                     
E   _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict                                           

<string>:124: PicklingError                                                                                                                 
________________________________ test_pickle[5] ________________________________                                                            

fd = F({'Sulla': 'Marco', 'Hicks': 'Bill', F({1: 2}): 'frozen'}), protocol = 5                                                              

>   ???                                                                                                                                     
E   _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict                                           

<string>:124: PicklingError                                                                                                                 
=========================== short test summary info ============================                                                            
FAILED test/test_frozendict_c.py::test_pickle[0] - TypeError: cannot pickle '...                                                            
FAILED test/test_frozendict_c.py::test_pickle[1] - TypeError: cannot pickle '...                                                            
FAILED test/test_frozendict_c.py::test_pickle[2] - _pickle.PicklingError: Can...                                                            
FAILED test/test_frozendict_c.py::test_pickle[3] - _pickle.PicklingError: Can...                                                            
FAILED test/test_frozendict_c.py::test_pickle[4] - _pickle.PicklingError: Can...                                                            
FAILED test/test_frozendict_c.py::test_pickle[5] - _pickle.PicklingError: Can...                                                            
FAILED test/test_frozendict_c.py::test_repr - AssertionError                                                                                
FAILED test/test_frozendict_c.py::test_str - AssertionError                                                                                 
FAILED test/test_frozendict_c.py::test_format - AssertionError                                                                              
FAILED test/test_frozendict_c_subclass.py::test_pickle[0] - TypeError: cannot...                                                            
FAILED test/test_frozendict_c_subclass.py::test_pickle[1] - TypeError: cannot...                                                            
FAILED test/test_frozendict_c_subclass.py::test_pickle[2] - _pickle.PicklingE...                                                            
FAILED test/test_frozendict_c_subclass.py::test_pickle[3] - _pickle.PicklingE...                                                            
FAILED test/test_frozendict_c_subclass.py::test_pickle[4] - _pickle.PicklingE...                                                            
FAILED test/test_frozendict_c_subclass.py::test_pickle[5] - _pickle.PicklingE...                                                            
================= 15 failed, 190 passed, 4 deselected in 0.38s =================                                                            
error: builder for '/nix/store/7anksnsb8jnl4pi4nr5mkh7kr1aqv459-python3.9-frozendict-2.1.0.drv' failed with exit code 1;                    
       last 10 log lines:                                                                                                                   
       > FAILED test/test_frozendict_c.py::test_repr - AssertionError                                                                       
       > FAILED test/test_frozendict_c.py::test_str - AssertionError                                                                        
       > FAILED test/test_frozendict_c.py::test_format - AssertionError                                                                     
       > FAILED test/test_frozendict_c_subclass.py::test_pickle[0] - TypeError: cannot...                                                   
       > FAILED test/test_frozendict_c_subclass.py::test_pickle[1] - TypeError: cannot...                                                   
       > FAILED test/test_frozendict_c_subclass.py::test_pickle[2] - _pickle.PicklingE...                                                   
       > FAILED test/test_frozendict_c_subclass.py::test_pickle[3] - _pickle.PicklingE...                                                   
       > FAILED test/test_frozendict_c_subclass.py::test_pickle[4] - _pickle.PicklingE...                                                   
       > FAILED test/test_frozendict_c_subclass.py::test_pickle[5] - _pickle.PicklingE...                                                   
       > ================= 15 failed, 190 passed, 4 deselected in 0.38s =================                                                   
       For full logs, run 'nix log /nix/store/7anksnsb8jnl4pi4nr5mkh7kr1aqv459-python3.9-frozendict-2.1.0.drv'.
risicle commented 2 years ago

@Marco-Sulla i'm increasingly thinking this might be our bug, possibly related to the funny stuff we have to do to make the macos sdk "reproducible" (as much as is possible).

Marco-Sulla commented 2 years ago

Yes, there are a lot of errors because I test every protocol of pickle. Currently the latest protocol is the number 5, so there are 6 pickle tests (it exists also the protocol zero), 6 tests for frozendict and 6 tests for a subclass of frozendict. The other 3 errors are trivial and can be ignored.

Can you do this test? You should open a terminal and do these commands:

git clone https://github.com/Marco-Sulla/python-frozendict.git cd python-frozendict python3 -m venv venv . venv/bin/activate pip install -U pip setuptools wheel pytest ./setup.py bdist_wheel pip install --force frozendict-2.1.0* cd test pytest

reckenrode commented 2 years ago

@Marco-Sulla i'm increasingly thinking this might be our bug, possibly related to the funny stuff we have to do to make the macos sdk "reproducible" (as much as is possible).

I’m not sure it’s just a Nix problem. I can also reproduce it using Python 3 from Xcode and frozendict installed via pip.

$ which python3
/usr/bin/python3
$ python3 -m venv venv
$ source venv/bin/activate.fish
(venv) reckenrode@dyrstelice ~ [0|1]> pip install --no-binary=frozendict frozendict
Collecting frozendict
  Downloading frozendict-2.1.0.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 2.7 MB/s 
Skipping wheel build for frozendict, due to binaries being disabled for it.
Installing collected packages: frozendict
    Running setup.py install for frozendict ... done
Successfully installed frozendict-2.1.0
WARNING: You are using pip version 20.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/Users/reckenrode/venv/bin/python3 -m pip install --upgrade pip' command.
(venv) reckenrode@dyrstelice ~> python3
Python 3.8.9 (default, Aug  3 2021, 19:21:54) 
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import frozendict
>>> import pickle
>>> pickle.dumps(frozendict.frozendict({1: 2}))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
>>> 
reckenrode commented 2 years ago

This is what I get using clang from Xcode 13.1.

$ clang --version
Apple clang version 13.0.0 (clang-1300.0.29.3)
Target: arm64-apple-darwin21.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
======================================================================================== short test summary info =========================================================================================
FAILED test_coold.py::test_pickle[0] - TypeError: cannot pickle 'dict' object
FAILED test_coold.py::test_pickle[1] - TypeError: cannot pickle 'dict' object
FAILED test_coold.py::test_pickle[2] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_coold.py::test_pickle[3] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_coold.py::test_pickle[4] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_coold.py::test_pickle[5] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_coold.py::test_repr - AssertionError
FAILED test_coold.py::test_str - AssertionError
FAILED test_coold.py::test_format - AssertionError
FAILED test_coold_subclass.py::test_pickle[0] - TypeError: cannot pickle 'dict' object
FAILED test_coold_subclass.py::test_pickle[1] - TypeError: cannot pickle 'dict' object
FAILED test_coold_subclass.py::test_pickle[2] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_coold_subclass.py::test_pickle[3] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_coold_subclass.py::test_pickle[4] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_coold_subclass.py::test_pickle[5] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_frozendict_c.py::test_pickle[0] - TypeError: cannot pickle 'dict' object
FAILED test_frozendict_c.py::test_pickle[1] - TypeError: cannot pickle 'dict' object
FAILED test_frozendict_c.py::test_pickle[2] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_frozendict_c.py::test_pickle[3] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_frozendict_c.py::test_pickle[4] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_frozendict_c.py::test_pickle[5] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_frozendict_c.py::test_repr - AssertionError
FAILED test_frozendict_c.py::test_str - AssertionError
FAILED test_frozendict_c.py::test_format - AssertionError
FAILED test_frozendict_c_subclass.py::test_pickle[0] - TypeError: cannot pickle 'dict' object
FAILED test_frozendict_c_subclass.py::test_pickle[1] - TypeError: cannot pickle 'dict' object
FAILED test_frozendict_c_subclass.py::test_pickle[2] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_frozendict_c_subclass.py::test_pickle[3] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_frozendict_c_subclass.py::test_pickle[4] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
FAILED test_frozendict_c_subclass.py::test_pickle[5] - _pickle.PicklingError: Can't pickle <class 'dict'>: it's not the same object as builtins.dict
===================================================================================== 30 failed, 287 passed in 0.41s =====================================================================================
$ otool -L ./frozendict/_frozendict.cpython-38-darwin.so
./frozendict/_frozendict.cpython-38-darwin.so (architecture x86_64):
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
./frozendict/_frozendict.cpython-38-darwin.so (architecture arm64):
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
Marco-Sulla commented 2 years ago

Ok, I have a suspect. The problem is I can't reproduce the bug, so I can't do experiments myself. What version of MacOS are you using?

reckenrode commented 2 years ago

I’m running macOS 12.0.1. I got the same results on an M1 Max as well as an x86_64 Mac. Xcode is the latest version from the App Store.

Edit: I also got the same results on an x86_64 Mac running macOS 11.6. I believe it also has Xcode 13.1 from the App Store. Interestingly, it does work with Python 3.10 from MacPorts. That’s interesting.

Edit 2: It also works using Python 3.10 from Nix. The version of Python in Xcode is 3.8. The default Python in nixpkgs-unstable is 3.9.

starcraft66 commented 2 years ago

I was using the latest version of macOS Monterey (12) in my testing.

Marco-Sulla commented 2 years ago

This is a problem. I don't have the latest version of MacOS. So I can't test it myself. On my Mojave it works. Could I commit on my repo what I think should be the solution, and someone can test it for me?

risicle commented 2 years ago

FWIW it was failing for me on macos 10.15, and hydra (our CI system https://hydra.nixos.org/eval/1724521?filter=frozendict&compare=1724426&full=) is running a mix of 10.15 and 10.14 (correct me?)

Marco-Sulla commented 2 years ago

I don't know what to say other than "works on my machine" :D

If you want I can try an alternative way to do __reduce__, using a tuple of items instead of a dict clone of frozendict as I do now. I can publish this code on my github repo, but I can't do an official release, because I'm not 100% sure this will fix something.

Marco-Sulla commented 2 years ago

Ok, I have Catalina now, and I can reproduce the bug with Python 3.8.2. I can try the workaround myself.

Marco-Sulla commented 2 years ago

It should be fixed in frozendict 2.1.1. Can you confirm?

reckenrode commented 2 years ago

It appears to build and work with Python 3.8 that’s bundled with Xcode as well as Python 3.9 from Nix, but it crashes during the tests using Python 3.8 from Nix. Here’s the build log.

Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing setuptools-build-hook
Using setuptoolsBuildPhase
Using setuptoolsShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
Sourcing setuptools-check-hook
Using setuptoolsCheckPhase
Sourcing pytest-check-hook
Using pytestCheckPhase
Removing setuptoolsCheckPhase
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/xvrxy3226g5ymq2yr783q1zlmgf4l7ii-frozendict-2.1.1.tar.gz
source root is frozendict-2.1.1
setting SOURCE_DATE_EPOCH to timestamp 1638139457 of file frozendict-2.1.1/setup.cfg
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
no configure script, doing nothing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
Executing setuptoolsBuildPhase
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.6-x86_64-3.8
creating build/lib.macosx-10.6-x86_64-3.8/frozendict
copying frozendict/__init__.py -> build/lib.macosx-10.6-x86_64-3.8/frozendict
copying frozendict/core.py -> build/lib.macosx-10.6-x86_64-3.8/frozendict
copying frozendict/VERSION -> build/lib.macosx-10.6-x86_64-3.8/frozendict
running build_ext
building 'frozendict._frozendict' extension
creating build/temp.macosx-10.6-x86_64-3.8
creating build/temp.macosx-10.6-x86_64-3.8/private
creating build/temp.macosx-10.6-x86_64-3.8/private/tmp
creating build/temp.macosx-10.6-x86_64-3.8/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0
creating build/temp.macosx-10.6-x86_64-3.8/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1
creating build/temp.macosx-10.6-x86_64-3.8/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict
creating build/temp.macosx-10.6-x86_64-3.8/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src
creating build/temp.macosx-10.6-x86_64-3.8/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8
creating build/temp.macosx-10.6-x86_64-3.8/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src
creating build/temp.macosx-10.6-x86_64-3.8/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src/Objects
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/Include -I/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src/Include -I/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src/Include/internal -I/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src/Objects -I/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src/Objects/stringlib -I/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src/Objects/clinic -I/nix/store/5rpfiqanhg1gi1a8k4qq69mxbhz3nxnq-python3-3.8.12/include/python3.8 -c /private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/frozendictobject.c -o build/temp.macosx-10.6-x86_64-3.8/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/frozendictobject.o -DPY_SSIZE_T_CLEAN -DPy_BUILD_CORE
clang-7: warning: argument unused during compilation: '-fno-strict-overflow' [-Wunused-command-line-argument]
/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/frozendictobject.c:2421:20: warning: incompatible pointer to integer conversion returning 'void *' from a function with result type 'Py_hash_t' (aka 'long') [-Wint-conversion]
            return NULL;
                   ^~~~
/nix/store/gsfxxazc51sx6vjdrz6cq8s19x7h5mwh-clang-wrapper-7.1.0/resource-root/include/stddef.h:105:16: note: expanded from macro 'NULL'
#  define NULL ((void*)0)
               ^~~~~~~~~~
/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/frozendictobject.c:297:1: warning: unused function 'dictkeys_incref' [-Wunused-function]
dictkeys_incref(PyDictKeysObject *dk)
^
2 warnings generated.
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/Include -I/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src/Include -I/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src/Include/internal -I/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src/Objects -I/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src/Objects/stringlib -I/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src/Objects/clinic -I/nix/store/5rpfiqanhg1gi1a8k4qq69mxbhz3nxnq-python3-3.8.12/include/python3.8 -c /private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src/Objects/dictobject.c -o build/temp.macosx-10.6-x86_64-3.8/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src/Objects/dictobject.o -DPY_SSIZE_T_CLEAN -DPy_BUILD_CORE
clang-7: warning: argument unused during compilation: '-fno-strict-overflow' [-Wunused-command-line-argument]
clang -bundle -undefined dynamic_lookup -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.11/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.6.0.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.4.1/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.2.5/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.20/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.36.0/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-6.3p08/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.1.1l/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.11/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.6.0.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.4.1/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.2.5/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.20/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.36.0/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-6.3p08/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.1.1l/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib build/temp.macosx-10.6-x86_64-3.8/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/frozendictobject.o build/temp.macosx-10.6-x86_64-3.8/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/frozendict/src/3_8/cpython_src/Objects/dictobject.o -L/nix/store/5rpfiqanhg1gi1a8k4qq69mxbhz3nxnq-python3-3.8.12/lib -o build/lib.macosx-10.6-x86_64-3.8/frozendict/_frozendict.cpython-38-darwin.so
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.11/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.6.0.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.4.1/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.2.5/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.20/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.36.0/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-6.3p08/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.1.1l/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.11/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.6.0.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.4.1/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.2.5/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.20/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.36.0/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-6.3p08/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.1.1l/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib'
installing to build/bdist.macosx-10.6-x86_64/wheel
running install
running install_lib
creating build/bdist.macosx-10.6-x86_64
creating build/bdist.macosx-10.6-x86_64/wheel
creating build/bdist.macosx-10.6-x86_64/wheel/frozendict
copying build/lib.macosx-10.6-x86_64-3.8/frozendict/__init__.py -> build/bdist.macosx-10.6-x86_64/wheel/frozendict
copying build/lib.macosx-10.6-x86_64-3.8/frozendict/core.py -> build/bdist.macosx-10.6-x86_64/wheel/frozendict
copying build/lib.macosx-10.6-x86_64-3.8/frozendict/VERSION -> build/bdist.macosx-10.6-x86_64/wheel/frozendict
copying build/lib.macosx-10.6-x86_64-3.8/frozendict/_frozendict.cpython-38-darwin.so -> build/bdist.macosx-10.6-x86_64/wheel/frozendict
running install_egg_info
running egg_info
writing frozendict.egg-info/PKG-INFO
writing dependency_links to frozendict.egg-info/dependency_links.txt
writing top-level names to frozendict.egg-info/top_level.txt
reading manifest file 'frozendict.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE.txt'
writing manifest file 'frozendict.egg-info/SOURCES.txt'
Copying frozendict.egg-info to build/bdist.macosx-10.6-x86_64/wheel/frozendict-2.1.1-py3.8.egg-info
running install_scripts
adding license file "LICENSE.txt" (matched pattern "LICEN[CS]E*")
creating build/bdist.macosx-10.6-x86_64/wheel/frozendict-2.1.1.dist-info/WHEEL
creating 'dist/frozendict-2.1.1-cp38-cp38-macosx_10_12_x86_64.whl' and adding 'build/bdist.macosx-10.6-x86_64/wheel' to it
adding 'frozendict/VERSION'
adding 'frozendict/__init__.py'
adding 'frozendict/_frozendict.cpython-38-darwin.so'
adding 'frozendict/core.py'
adding 'frozendict-2.1.1.dist-info/LICENSE.txt'
adding 'frozendict-2.1.1.dist-info/METADATA'
adding 'frozendict-2.1.1.dist-info/WHEEL'
adding 'frozendict-2.1.1.dist-info/top_level.txt'
adding 'frozendict-2.1.1.dist-info/RECORD'
removing build/bdist.macosx-10.6-x86_64/wheel
Finished executing setuptoolsBuildPhase
@nix { "action": "setPhase", "phase": "installPhase" }
installing
Executing pipInstallPhase
/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/dist /private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1
Processing ./frozendict-2.1.1-cp38-cp38-macosx_10_12_x86_64.whl
Installing collected packages: frozendict
Successfully installed frozendict-2.1.1
/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1
Finished executing pipInstallPhase
@nix { "action": "setPhase", "phase": "fixupPhase" }
post-installation fixup
strip is /nix/store/bx4prwlqjf856w4swv6vdz7v9avhwixy-cctools-binutils-darwin-949.0.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/nd0b93y9rxxj5b4pbm09bvyyhr2ivj61-python3.8-frozendict-2.1.1/lib 
patching script interpreter paths in /nix/store/nd0b93y9rxxj5b4pbm09bvyyhr2ivj61-python3.8-frozendict-2.1.1
Executing pythonRemoveTestsDir
Finished executing pythonRemoveTestsDir
@nix { "action": "setPhase", "phase": "installCheckPhase" }
running install tests
no Makefile or custom installCheckPhase, doing nothing
@nix { "action": "setPhase", "phase": "pythonCatchConflictsPhase" }
pythonCatchConflictsPhase
@nix { "action": "setPhase", "phase": "pythonRemoveBinBytecodePhase" }
pythonRemoveBinBytecodePhase
@nix { "action": "setPhase", "phase": "pythonImportsCheckPhase" }
pythonImportsCheckPhase
Executing pythonImportsCheckPhase
Check whether the following modules can be imported: frozendict
@nix { "action": "setPhase", "phase": "pytestCheckPhase" }
pytestCheckPhase
Executing pytestCheckPhase
============================= test session starts ==============================
platform darwin -- Python 3.8.12, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1
collecting ... 
collected 209 items / 4 deselected / 205 selected                              

test/test_frozendict.py ................................................ [ 23%]
....                                                                     [ 25%]
test/test_frozendict_c.py .............................................. [ 47%]
.......                                                                  [ 51%]
test/test_frozendict_c_subclass.py ........................Fatal Python error: Segmentation fault

Current thread 0x0000000204d61600 (most recent call first):
  File "/private/tmp/nix-build-python3.8-frozendict-2.1.1.drv-0/frozendict-2.1.1/test/test_frozendict_c_subclass.py", line 6 in __new__
  File "<string>", line 138 in test_pickle
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/python.py", line 1641 in runtest
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/runner.py", line 311 in from_call
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/runner.py", line 254 in call_runtest_hook
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/main.py", line 323 in _main
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/nix/store/1216wh5zl0aihlllllfsf49p6wvr5wzk-python3.8-pluggy-1.0.0/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/config/__init__.py", line 162 in main
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "/nix/store/jgx91xjidlp66bjhj7r97sg47sipr3p0-python3.8-pytest-6.2.5/lib/python3.8/site-packages/pytest/__main__.py", line 5 in <module>
  File "/nix/store/5rpfiqanhg1gi1a8k4qq69mxbhz3nxnq-python3-3.8.12/lib/python3.8/runpy.py", line 87 in _run_code
  File "/nix/store/5rpfiqanhg1gi1a8k4qq69mxbhz3nxnq-python3-3.8.12/lib/python3.8/runpy.py", line 194 in _run_module_as_main
/nix/store/p29idq7mnl2wnq144bdg8k7cqg8ygg4i-pytest-check-hook/nix-support/setup-hook: line 53: 45522 Segmentation fault: 11  /nix/store/5rpfiqanhg1gi1a8k4qq69mxbhz3nxnq-python3-3.8.12/bin/python3.8 -m pytest -k "not test_union" --ignore="test/test_coold.py" --ignore="test/test_coold_subclass.py"
reckenrode commented 2 years ago

The stack trace is in pytest, so I wonder if the problem is now on the Nix side of things.

Marco-Sulla commented 2 years ago

Could you run ./test/debug.py 100?

reckenrode commented 2 years ago

Could you run ./test/debug.py 100?

This is what I get with Python 3.8 from Nix:

frozendict_class(dict_1)
frozendict_class(dict_3)
frozendict_class()
frozendict_class({})
frozendict_class([])
frozendict_class({}, **{})
frozendict_class(**dict_1)
frozendict_class(dict_1, **dict_2)
frozendict_class(fd_1)
fd_1.copy()
fd_1 == dict_1
fd_1 == fd_1
pickle.loads(pickle.dumps(fd_1))
frozendict_class(dict_1_items)
fd_1.keys()
fd_1.values()
fd_1.items()
frozendict_class.fromkeys(dict_1)
frozendict_class.fromkeys(dict_1, 1)
frozendict_class.fromkeys(dict_1_keys)
frozendict_class.fromkeys(dict_1_keys, 1)
frozendict_class.fromkeys(dict_1_keys_set)
frozendict_class.fromkeys(dict_1_keys_set, 1)
repr(fd_1)
fd_1 | dict_2
hash(fd_1)

for x in fd_1:
    pass

for x in fd_1.keys():
    pass

for x in fd_1.values():
    pass

for x in fd_1.items():
    pass

try:
    hash(fd_unashable)
except TypeError:
    pass

frozendict_class(dict_1)
frozendict_class(dict_3)
frozendict_class()
frozendict_class({})
frozendict_class([])
frozendict_class({}, **{})
frozendict_class(**dict_1)
frozendict_class(dict_1, **dict_2)
frozendict_class(fd_1)
fd_1.copy()
fd_1 == dict_1
fd_1 == fd_1
pickle.loads(pickle.dumps(fd_1))
frozendict_class(dict_1_items)
fd_1.keys()
fd_1.values()
fd_1.items()
frozendict_class.fromkeys(dict_1)
frozendict_class.fromkeys(dict_1, 1)
frozendict_class.fromkeys(dict_1_keys)
frozendict_class.fromkeys(dict_1_keys, 1)
frozendict_class.fromkeys(dict_1_keys_set)
frozendict_class.fromkeys(dict_1_keys_set, 1)
repr(fd_1)
fd_1 | dict_2
hash(fd_1)

for x in fd_1:
    pass

for x in fd_1.keys():
    pass

for x in fd_1.values():
    pass

for x in fd_1.items():
    pass

try:
    hash(fd_unashable)
except TypeError:
    pass
reckenrode commented 2 years ago

I’m getting a similar crash building the ledger-autosync derivation, so I don’t think the remaining crash problem is unlikely a frozendict problem.

Marco-Sulla commented 2 years ago

May you re-test with the latest version? I completely refactored the C code. My CI/CD gives me no segfaults now, with more than 600 tests positive on MacOS, Linux and Windows.

uri-canva commented 2 years ago

I'm not sure if I did it correctly, pip install --force frozendict-2.1.0* in the original instructions didn't work for me, so I did pip install ., but all the tests pass:

(venv) bash-5.1$ python --version
Python 3.9.9
(venv) bash-5.1$ clang --version
Apple clang version 12.0.5 (clang-1205.0.22.11)
Target: x86_64-apple-darwin21.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
(venv) bash-5.1$ sw_vers
ProductName:    macOS
ProductVersion: 12.0.1
BuildVersion:   21A559
Marco-Sulla commented 2 years ago

Yes, because now the version is 2.2.0. Thank you for the report :)