search

Quansight / omnisci

Explorations on using MapD and Jupyter together.
4 stars 1 forks source link

Conda package for omnisci and omnisci-tools #25

Closed dharhas closed 5 years ago

dharhas commented 5 years ago

Finish conda packaging workflow. Goal is to get to 2 conda packages that allow for the following - 'conda install omnisci' to build/install the open source core, and 'conda install

omnisci-tools' to install the ibis, jupyterlab-omnisci, altair tools. With this, a user should be able to run omnisci locally and analyze data using jupyterlab + omnisci + ibis + altair

pearu commented 5 years ago

Status of omnisci-core (former mapd-core) conda package on OSX:

Status of omnisci-tools:

pearu commented 5 years ago

tensorflow conda UX as an example of GPU enabled conda packaging: https://towardsdatascience.com/tensorflow-gpu-installation-made-easy-use-conda-instead-of-pip-52e5249374bc

pearu commented 5 years ago

Notes for omnisci-core build in conda environment with GPU enabled:

pearu commented 5 years ago

mac osx conda build status update:

Label Time Summary: sanity = 148.59 sec*proc (17 tests)

Total Test time (real) = 148.65 sec

The following tests FAILED: 2 - UpdelStorageTest (Failed) 6 - ExecuteTest (Failed) 15 - TopKTest (Failed)

- debugging with lldb:

sudo lldb Tests/ExecuteTest (lldb) run ...... [----------] 115 tests from Select [ RUN ] Select.NullGroupBy Process 11588 stopped

xmnlab commented 5 years ago

@pearu .. not sure if it could help .. but maybe the conf cpu-buffer-mem-bytes = 1000000000 could help here: https://github.com/ibis-project/ibis/blob/master/ci/omnisci.conf#L3

@andrewseidl do you think it could help? or do you have any other thoughts?

pearu commented 5 years ago

@xmnlab note that the error is obtained in mac mini box.

andrewseidl commented 5 years ago

Is this failing with CUDA enabled or disabled?

@xmnlab shouldn't need to set cpu-buffer-mem-bytes, by default it will be allowed to use up to 80% of the system mem.

@pearu could you try running with the GLOG_logtostderr=1 env var set? Just to get a bit more debug info.

pearu commented 5 years ago

@andrewseidl This is CUDA disabled build. With GLOG_logtostderr=1:

[ RUN      ] Select.NullGroupBy
I0528 10:16:02.426483 400070080 Calcite.cpp:300] Time to updateMetadata 0 (ms)
...
I0528 10:16:02.427465 400070080 Catalog.cpp:1234] Instantiating Fragmenter for table table_null_group_by took 0ms
I0528 10:16:02.432482 400070080 Catalog.cpp:1303] Time to load Dictionary 1_96 was 4ms
I0528 10:16:02.962232 400070080 Calcite.cpp:432] User mapd catalog mapd sql 'SELECT val FROM table_null_group_by GROUP BY val;'
I0528 10:16:02.970645 400070080 Calcite.cpp:452] Time in Thrift 1 (ms), Time in Java Calcite server 7 (ms)
...
I0528 10:16:02.982969 400070080 Calcite.cpp:300] Time to updateMetadata 1 (ms)
...
I0528 10:16:02.987901 400070080 Calcite.cpp:300] Time to updateMetadata 0 (ms)
...
I0528 10:16:02.988775 400070080 Catalog.cpp:1234] Instantiating Fragmenter for table table_null_group_by took 0ms
I0528 10:16:03.517801 400070080 Calcite.cpp:432] User mapd catalog mapd sql 'SELECT val FROM table_null_group_by GROUP BY val;'
I0528 10:16:03.524819 400070080 Calcite.cpp:452] Time in Thrift 1 (ms), Time in Java Calcite server 5 (ms)
2019-05-28 10:16:03 ERROR TThreadPoolServer:run:315 - Thrift error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
        ... 9 more
E0528 10:16:03.556411 30892032 QueryRunner.cpp:113] Interrupt signal (11) received.
I0528 10:16:03.556426 30892032 Calcite.cpp:512] Shutting down Calcite server
I0528 10:16:03.560261 30892032 Calcite.cpp:521] shut down Calcite
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0528 10:16:03.567929 30892032 Calcite.cpp:527] Destroy Calcite Class
I0528 10:16:03.567950 30892032 Calcite.cpp:529] End of Calcite Destructor 
<end of output>

where ... are typical messages with ERROR TThreadPoolServer:run:315 - Thrift error occurred during processing of message.

pearu commented 5 years ago

The crash happens when calling

child.wait();

in Executor::dispatchFragments. Before calling wait, child.valid() returns true.

pearu commented 5 years ago

@andrewseidl , I have tracked the crash down to QueryExecutionContext::launchCpuCode where literal_buff[0] is evaluated while literal_buff.size() is 0.

pearu commented 5 years ago

Interestingly, in Linux (using gcc) literal_buff[0] does not cause a crash even when literal_buff.size() == 0.

pearu commented 5 years ago

@andrewseidl , I think we have hit a compiler-dependent issue described in https://stackoverflow.com/questions/3829788/using-operator-on-empty-stdvector

pearu commented 5 years ago

Defining

#define GETVECTORPTR(A) ((A.size()>0) ? &A[0] : nullptr)

and wrapping all &A[0] usages with GETVECTORPTR fixes some of test crashes in Mac OSX.

pearu commented 5 years ago

The issue of conda packaging omnscidb-cpu for Mac OSX with test failures has a solution: https://github.com/conda-forge/omniscidb-cpu-feedstock/pull/7

pearu commented 5 years ago

On the following build failure:

/home/pearu/miniconda3/envs/omnisci-gpu-dev/include/llvm/Object/SymbolicFile.h:48:31: error: expected ')' before 'PRIxPTR'

use

export CXXFLAGS="$CXXFLAGS -D__STDC_FORMAT_MACROS=1"

as a fix.

gnestor commented 5 years ago

omnisci-pytools is available on conda-forge: https://github.com/conda-forge/omnisci-pytools-feedstock

conda install omnisci-pytools will install jupyterlab-omnisci and jupyter-widgets JupyterLab extensions and install all dependencies.

pearu commented 5 years ago

Moved the remaining task to a separate issue (#63 ).