search

capnproto / pycapnp

Cap'n Proto serialization/RPC system - Python bindings
BSD 2-Clause "Simplified" License
466 stars 124 forks source link

KjException: Duplicate ID when mixing absolute and relative imports #62

Open breznak opened 9 years ago

breznak commented 9 years ago

hi,

when I use pyCAPNP along with cProfile module, I get the error. (alone both work fine)

  File "enc_speed.py", line 28, in <module>
    from nupic.encoders.scalar import ScalarEncoder
  File "nupic/encoders/__init__.py", line 34, in <module>
    from multi import MultiEncoder
  File "nupic/encoders/multi.py", line 40, in <module>
    from nupic.encoders.scalar_capnp import ScalarEncoderProto
  File "capnp/lib/capnp.pyx", line 3773, in capnp.lib.capnp._Loader.load_module (capnp/lib/capnp.cpp:71749)
  File "capnp/lib/capnp.pyx", line 3752, in capnp.lib.capnp.load (capnp/lib/capnp.cpp:71239)
  File "capnp/lib/capnp.pyx", line 3126, in capnp.lib.capnp.SchemaParser.load (capnp/lib/capnp.cpp:63334)
capnp.lib.capnp.KjException: /home/mmm/devel/nupic/nupic-source/nupic/encoders/scalar.capnp:0: failed: Duplicate ID @0xfa7d16f86048a6e4.
stack: 0x7f848fcd93f8 0x7f848fbfb329 0x7f848fbfae75 0x7f848fc20e6a 0x7f848fc213a2 0x7f848fc25796 0x7f848fc2588b 0x7f848fc25de9 0x7f848fc25e28 0x7f848fbfa29e 0x7f848fbfa38b 0x7f848fbaab98 0x7f848fb9ab96 0x7f848fb7c23f 0x7f848fb4f54f 0x7f848fb74bca

versions:

For reference, please see https://github.com/numenta/nupic/issues/2067

scottpurdy commented 9 years ago

I received a similar error elsewhere [1] without using cProfile. It appears to only be an issue with absolute imports (which we would really much prefer to use) and even then only in some circumstances. I saw the issue for both 0.5.5 and 0.5.6.

[1] https://github.com/numenta/nupic/issues/2070

dioptre commented 9 years ago

+1 :(

Latest ugrades to pycapnp and nupic 39571ee12110

jparyani commented 9 years ago

Sorry, I've been looking into this issue the past few days. I'm able to reproduce, but having trouble tracking down the root of the problem.

scottpurdy commented 9 years ago

@jparyani - No rush! And if you don't have time now just let me know and we will go back to relative imports for now to avoid the issue.

jparyani commented 9 years ago

@scottpurdy I'd recommend switching to relative imports for now. My best guess so far is that you're using a mix of relative and absolute import paths, which should theoretically work fine together, but in practice could be causing this issue.

scottpurdy commented 9 years ago

@jparyani - Yes, mix of relative and absolute. Do you want to change the title of this issue to reflect that and track progress here?

@oxtopus - fyi

david-ragazzi commented 9 years ago

I had this same problem (i.e. Duplicate ID) but because another reason. I realized that I copied and pasted a existent struct but forgot change the name.. hehe 

struct MyStruct1 {
 ..
}

# Different struct but with same name of the previous one
struct MyStruct1 {
 ...
}
scottpurdy commented 9 years ago

@jparyani - This is biting us again as we move things around (related to passing capnp structs from C++ to Python). We no longer have any absolute imports though. We build a dynamic library that can be loaded by Python (using SWIG). This library has some .capnp schema files compiled into it. Then in Python, we use the import hooks to directly import .capnp files, some of which are compiled into the dynamic library that is loaded into the process.

We are seeing the duplicate id error in some cases and I am guessing that it is because the schema using the id was compiled into the dynamic library and is also being loaded via the Python .capnp import.

It would be very limiting for us to not be able to include the same .capnp schema in the C++ library used by Python and also directly from the Python code. Is this a bug in pycapnp or a more funamental issue with loading a library that a schema was compiled into and then separating using the pycapnp import for the same schema?

CC @akhilaananthram

jparyani commented 9 years ago

Is this a bug in pycapnp or a more funamental issue with loading a library that a schema was compiled into and then separating using the pycapnp import for the same schema?

This definitely should be possible. loadCompiledTypeAndDependencies was made specifically to bridge the two, and it doesn't seem to be the place where this error is occurring (it's occurring in the compiler's parsing stage).

If it's not too much trouble, could you make a minimal reproduction of the problem? Once you do, I'll try and repro it in the upstream Cap'n Proto C++ library.

akhilaananthram commented 9 years ago

I am working on a simple example with the same problem.

I played around with this in the Python shell and in the file. In the Python shell, it works if I import capnp first. This didn't help in the file. I also tried calling capnp.load to see if I could load the file that way if I give it a path to the file. That works in the shell, but didn't work in the file.

akhilaananthram commented 9 years ago

Actually, we solved the problem. It looks like one of our import paths was pointing to an older version of a capnp file. Once we changed that, we no longer get a duplicate ID error.

scottpurdy commented 9 years ago

This issue seems to be coming up again when people update to Python 2.7.10. Here is one example and we also see this on our internal CI pipelines when they were updated to 2.7.10. I'll see if I can create a simple way to reproduce it, although I thought you were able to reproduce at least one of the issues?

jparyani commented 9 years ago

I'm having a lot of trouble reproducing it today (I think due to some other failure in the tests). When running $NUPIC/scripts/run_nupic_tests -u --coverage, the tests seem to fail with:

============================================================================== test session starts ===============================================================================
platform darwin -- Python 2.7.9 -- pytest-2.4.2 -- /Users/jason/.virtualenvs/nupic/bin/python2.7
plugins: cov, xdist
collecting 6 items / 1 errors
===================================================================================== ERRORS =====================================================================================
__________________________________________________ ERROR collecting tests/unit/nupic/algorithms/anomaly_likelihood_jeff_test.py __________________________________________________
tests/unit/nupic/algorithms/anomaly_likelihood_jeff_test.py:36: in <module>
    from nupic.algorithms import anomaly_likelihood as an
nupic/algorithms/__init__.py:28: in <module>
    from nupic.bindings.math import NearestNeighbor
E   ImportError: No module named math
tomoprime commented 9 years ago

jparyani you are not running python 2.7.10. Note if you update your mac OS X10.10.5 your python gets upgraded automatically.

scottpurdy commented 9 years ago

It looks like the C extensions for NuPIC aren't installed. I'm not sure which version of NuPIC you're on though. I'll try to see what the easiest way to reproduce it is, although I'm not sure if the 2.7.10 issue is the same as the issue when mixing absolute and relative paths.

jparyani commented 9 years ago

I'll update to OSX 10.10.5 tonight and report back.

On the version issue, something really weird happens and I'm getting the 10.9 version instead of 10.10 (I'm on 10.10.4):

pip install nupic
Collecting nupic
  Using cached nupic-0.2.6-cp27-none-macosx_10_9_intel.whl
...

It may be due to using homebrew's Python instead of the system installed Python.

tomoprime commented 9 years ago

I'd brew uninstall python. I had done so after updating OS X. Note they are currently rebuilding nupic after merging all the PRs. A new release is on the way...

jparyani commented 9 years ago

I'm still not able to get the tests to run to reproduce this issue. Installing nupic==0.2.8 (the latest) fails with:

Collecting nupic.bindings==0.1.0 (from nupic)
  Could not find a version that satisfies the requirement nupic.bindings==0.1.0 (from nupic) (from versions: )
No matching distribution found for nupic.bindings==0.1.0 (from nupic)

https://github.com/numenta/nupic/issues/2357 seems to be tracking this issue..

Installing 0.2.6 and running the tests fails with a new error this time:

============================================================================== test session starts ===============================================================================
platform darwin -- Python 2.7.10 -- pytest-2.4.2 -- /Users/jason/.virtualenvs/main/bin/python
plugins: cov, xdist
collecting 117 items / 1 errors
===================================================================================== ERRORS =====================================================================================
____________________________________________________________ ERROR collecting tests/unit/nupic/data/dictutils_test.py ____________________________________________________________
tests/unit/nupic/data/dictutils_test.py:28: in <module>
    from nupic.swarming.hypersearch.utils import rCopy
E   ImportError: No module named hypersearch.utils
scottpurdy commented 9 years ago

@jparyani - Alright I have some new info and a way that should allow you to reproduce (at end). First, new info:

  1. It looks like when I do "editable"/"develop" installations of our Python packages that I do not see the Duplicate ID error. I only get the error when I do a regular "python setup.py install"
  2. It looks like the errors are happening when I run my test suite under py.test but not when I run an individual test directly. I tried the --boxed option under py.test but this still resulted in errors. Even running a single offending test via py.test (as opposed to whole test suite) exhibits the failure.
  3. I tried putting some debugging statements in pycapnp source (on top of 0.5.5 version that we are using) but this didn't reveal anything that stood out. It looks like the full paths to the .capnp files are passed on to the core capnp library so the error must be coming from there.

I'm not sure what to make of the above points.

Now, a way that you should be able to replicate by doing source installs of nupic.core and then nupic. Sorry this isn't more straight-forward, our build system is in the midst of some changes.

brando90 commented 3 years ago

I also had this issue:

Connected to pydev debugger (build 211.6693.115)
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/brando/ML4Coq/ml4coq-proj/data_lib/dag/dag_dataloader.py", line 23, in <module>
    from data_lib.dag.dataset_preparation import DagDataPreparation, process_dag_dataset_to_produce_splits_and_indexing
  File "/Users/brando/ML4Coq/ml4coq-proj/data_lib/dag/dataset_preparation.py", line 36, in <module>
    import dag_api_capnp
ModuleNotFoundError: No module named 'dag_api_capnp'

However, I managed to solve it. This is what I did (thanks for the part discussions!):

  1. I made sure that all paths to my captain proto schema were pointing to the same one
  2. I made sure that all paths to my captain proto schema were absolute

Since I have different repos using the schema the best solution that was consistent throughout both code bases was to use absolute schema's. This seemed to have fixed the issue. Some were pointing to an old schema which I believe is the most likely source of the bug. It would be nice to know if we can mix relative and absolute. It would also be nice to get a more descriptive error message? I would not have known the issue was that they were pointing to different schemas without reading this thread.

Thanks!

Code that made it work:

# import dag_api_capnp
capnp.remove_import_hook()
# dag_api_capnp = str(Path('~/data_lib/dag/dag_api.capnp').expanduser())
dag_api_capnp = str(Path('~/src/dag_api.capnp').expanduser())
dag_api_capnp = capnp.load(dag_api_capnp)