search

dsp-uga / elizabeth

Scalable malware detection
MIT License
0 stars 0 forks source link

Convolutional neural net with BigDL #29

Closed cbarrick closed 1 year ago

cbarrick commented 6 years ago

I don't know if I'm going to finish this in time.

TODO

cbarrick commented 6 years ago

Currently I get this very mysterious error:

Traceback (most recent call last):
  File "/tmp/4d4466f9-3bf2-47ef-a26f-bbf80e3db8d4/driver.py", line 4, in <module>
    elizabeth.__main__.main()
  File "/tmp/4d4466f9-3bf2-47ef-a26f-bbf80e3db8d4/elizabeth-0.1.0.dev0-py3.6.egg/elizabeth/__main__.py", line 53, in main
  File "/tmp/4d4466f9-3bf2-47ef-a26f-bbf80e3db8d4/elizabeth-0.1.0.dev0-py3.6.egg/elizabeth/cnn.py", line 58, in main
  File "/opt/conda/lib/python3.6/site-packages/bigdl/nn/layer.py", line 1092, in __init__
    super(Sequential, self).__init__(None, bigdl_type)
  File "/opt/conda/lib/python3.6/site-packages/bigdl/nn/layer.py", line 636, in __init__
    super(Container, self).__init__(jvalue, bigdl_type, *args)
  File "/opt/conda/lib/python3.6/site-packages/bigdl/nn/layer.py", line 70, in __init__
    bigdl_type, JavaValue.jvm_class_constructor(self), *args)
  File "/opt/conda/lib/python3.6/site-packages/bigdl/util/common.py", line 549, in callBigDlFunc
    jinstance = JavaCreator.instance(bigdl_type=bigdl_type).value
  File "/opt/conda/lib/python3.6/site-packages/bigdl/util/common.py", line 53, in instance
    cls._instance = cls(bigdl_type)
  File "/opt/conda/lib/python3.6/site-packages/bigdl/util/common.py", line 75, in __init__
    self.value = getattr(jclass, "ofFloat")()
TypeError: 'JavaPackage' object is not callable

I know there are other problems down the line, but I can't even get to those errors until I pass this.

It seems like an error in BigDL...

cbarrick commented 6 years ago

Initializing the engine only moves the source of the error to the init_engine() call

cbarrick commented 6 years ago

This seems to be an issue of getting Spark to find the BigDL jars. The fix will likely involve updates to the deployment scripting rather than the source code.

cbarrick commented 1 year ago

Closing this to get it out of my queue