`kgtk graph-embeddings` fails in bigtorchgraph with a subprocessing error

[CraigMiloRogers]

@szeke reports:

!kgtk graph-embeddings -i $OUT/arnold.family.tsv -o $OUT/arnold.family.embeddings.tsv \
--dimension 30

INFO:torchbiggraph:Loading entity counts...
INFO:torchbiggraph:Creating workers...
INFO:torchbiggraph:Initializing global model...
INFO:torchbiggraph:Starting epoch 1 / 100, edge path 1 / 1, edge chunk 1 / 1
INFO:torchbiggraph:Edge path: /tmp/output/edges_partitioned
INFO:torchbiggraph:still in queue: 0
INFO:torchbiggraph:Swapping partitioned embeddings None ( 0 , 0 )
INFO:torchbiggraph:( 0 , 0 ): Loading entities
A subprocess exited unexpectedly with status -11

[CraigMiloRogers]

The graph-embedding code ran OK for me when I ran it isolated on a command line. Inside your Python notebook it failed for me with 'AF_UNIX Path too long`. This isn't the same error you got, but I suspect a similar cause: defining the envar TEMP to something very long. TEMP is used deep down in Python's standard library for such things as naming the files used for AF_UNIX interprocess communication to implement the multiprocessing thread pool. In this context the file name limit is 108 characters on Linux, and I think shorter on MacOS. The solution is to define KGTK_TEMP instead of TEMP in configure_kgtk(...).

[CraigMiloRogers]

https://docs.python.org/3/library/tempfile.html

The default directory is chosen from a platform-dependent list, but the user of the application can control the directory location by setting the TMPDIR, TEMP or TMP environment variables. 

[CraigMiloRogers]

I don't find this documented in any obvious top-level location, such as:

https://docs.python.org/3/using/cmdline.html

I wonder how many other environment variables are buried in Python's standard library?

[szeke]

I ran outside Jupyter with the KGTK_TEMP variable and get the same error:

(kgtk-env) macbook-pro:tutorial-network-analysis pedroszekely$ kgtk graph-embeddings -i arnold.family.tsv -o arnold.family.embeddings.txt --dimension 30
INFO:torchbiggraph:Loading entity counts...
INFO:torchbiggraph:Creating workers...
INFO:torchbiggraph:Initializing global model...
INFO:torchbiggraph:Starting epoch 1 / 100, edge path 1 / 1, edge chunk 1 / 1
INFO:torchbiggraph:Edge path: /tmp/output/edges_partitioned
INFO:torchbiggraph:still in queue: 0
INFO:torchbiggraph:Swapping partitioned embeddings None ( 0 , 0 )
INFO:torchbiggraph:( 0 , 0 ): Loading entities
A subprocess exited unexpectedly with status -11
(kgtk-env) macbook-pro:tutorial-network-analysis pedroszekely$ ls -l /Users/pedroszekely/tmp
(kgtk-env) macbook-pro:tutorial-network-analysis pedroszekely$ echo $KGTK_TEMP
/Users/pedroszekely/tmp

[szeke]

@saggu Please give it a try on your machine. It is the last cell in https://github.com/usc-isi-i2/kgtk/blob/dev/tutorial/kg-network-analysis.ipynb

[saggu]

@szeke I can run this without errors. Trying now with a fresh kgtk virtual env

[saggu]

No issues running with a fresh env as well,

python --version = Python 3.8.12 conda --version = conda 4.7.12

gcc --version

Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.5 (clang-1205.0.22.11)
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[Bin-Go2]

Hi, I am sorry about the late reply. I have graduated from USC and am now in China.  Today I just checked my email and get the feedback on the kgtk-graph embedding issue. I suppose Jerry Qiu is working on this module right now if my memory is correct. Maybe you can ask him for help. :) 

Configuration of the environment sometimes may cause unknown bugs. Please keep an eye on it first.