KeyError when running

[vkoppaka]

Hi there,

I have been trying to install Thug into a Google Container Engine (Kubernetes) to process some URLs from a queue. The installation seems to go well without any issues but when I try to use Thug (via ThugAPI), I get a 'KeyError winxipie60'. I get this error even if I force another user agent using set_useragent method.

Here is the traceback KeyError: 'winxpie60' return self[log.ThugOpts.useragent]['browserTag'].startswith('ie') File "/env/local/lib/python2.7/site-packages/thug/DOM/Personality.py", line 85, in isIE if log.ThugOpts.Personality.isIE(): File "/env/local/lib/python2.7/site-packages/thug/DOM/W3C/Events/EventTarget.py", line 17, in __init_personality self.__init_personality() File "/env/local/lib/python2.7/site-packages/thug/DOM/W3C/Events/EventTarget.py", line 14, in __init__ EventTarget.__init__(self) File "/env/local/lib/python2.7/site-packages/thug/DOM/W3C/Node.py", line 33, in __init__ Node.__init__(self, doc) File "/env/local/lib/python2.7/site-packages/thug/DOM/W3C/Document.py", line 25, in __init__ Document.__init__(self, doc) File "/env/local/lib/python2.7/site-packages/thug/DOM/W3C/HTML/HTMLDocument.py", line 33, in __init__ return DOMImplementation(BeautifulSoup.BeautifulSoup(html, "html.parser"), **kwds) File "/env/local/lib/python2.7/site-packages/thug/DOM/W3C/w3c.py", line 13, in parseString doc = w3c.parseString('') File "/env/local/lib/python2.7/site-packages/thug/ThugAPI/ThugAPI.py", line 391, in run_remote self.run_remote(url) File "/home/vmagent/app/thug_analyzer.py", line 33, in analyze analyzer.analyze(url) File "/app/main.py", line 33, in analyze_with_thug result = self.fn(*self.args, **self.kwargs) File "/env/local/lib/python2.7/site-packages/concurrent/futures/thread.py", line 62, in run return self.__get_result() File "/env/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result logger.debug('Result: %s', future.result()) File "/env/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/subscriber/policy/thread.py", line 36, in _callback_completed callback(self) File "/env/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 301, in _invoke_callbacks Traceback (most recent call last): [2017-11-27 16:48:45] exception calling callback for <Future at 0x7f36d4236ed0 state=finished raised KeyError> [2017-11-27 16:48:45] [CRITICAL] Logging subsystem not initialized (configuration file not found) [2017-11-27 16:48:45] [WARNING] VirusTotal disabled (no configuration file found) [2017-11-27 16:48:45] [Text Classifier] Skipping not existing default filter file [2017-11-27 16:48:45] [Text Classifier] Skipping not existing default classification rule file [2017-11-27 16:48:45] [Sample Classifier] Skipping not existing default filter file [2017-11-27 16:48:45] [Sample Classifier] Skipping not existing default classification rule file [2017-11-27 16:48:45] [URL Classifier] Skipping not existing default filter file [2017-11-27 16:48:45] [URL Classifier] Skipping not existing default classification rule file [2017-11-27 16:48:45] [VBS Classifier] Skipping not existing default filter file [2017-11-27 16:48:45] [VBS Classifier] Skipping not existing default classification rule file [2017-11-27 16:48:45] [JS Classifier] Skipping not existing default filter file [2017-11-27 16:48:45] [JS Classifier] Skipping not existing default classification rule file [2017-11-27 16:48:45] [HTML Classifier] Skipping not existing default filter file [2017-11-27 16:48:45] [HTML Classifier] Skipping not existing default classification rule file

And here is the Dockerfile

` FROM gcr.io/google_appengine/python

RUN virtualenv -p python2.7 /env

ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH

RUN apt-get update && \
  apt-get install -y --no-install-recommends \
    build-essential \
    python-dev \
    python-setuptools \
    libboost-python-dev \
    libboost-all-dev \
    python-pip \
    libxml2-dev \
    libxslt-dev \
    git \
    libtool \
    graphviz-dev \
    automake \
    libffi-dev \
    graphviz \
    libfuzzy-dev \
    libjpeg-dev \
    pkg-config \
    autoconf && \
  rm -rf /var/lib/apt/lists/*

RUN easy_install -U setuptools pygraphviz==1.3.1

USER root
WORKDIR /home

RUN git clone https://github.com/thug/pyv8.git && \
  ls && \
  cd pyv8 && \
  python setup.py build && \
  python setup.py install && \
  cd .. && \
  rm -rf pyv8

RUN pip install thug

RUN groupadd -r thug && \
  useradd -r -g thug -d /home/thug -s /sbin/nologin -c "Thug User" thug && \
  mkdir -p /home/thug /tmp/thug/logs && \
  chown -R thug:thug /home/thug /tmp/thug/logs

RUN echo "/opt/libemu/lib/" > /etc/ld.so.conf.d/libemu.conf && ldconfig

ADD requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt

ADD . /app

CMD honcho start -f /app/Procfile $PROCESSES`

And here is my requirements file.

google-cloud==0.29.0 google-cloud-pubsub==0.29.0 gunicorn==19.7.1 honcho==1.0.1 psq==0.5.0 beautifulsoup4==4.6.0 html5lib==0.999999999 lxml==4.1.1 cchardet==2.1.1 requests==2.18.4 PySocks==1.6.7 cssutils==1.0.2 zope.interface==4.4.3 pygraphviz==1.3.1 python-magic==0.4.13 rarfile==3.0 networkx==2.0 pymongo==3.5.1 ssdeep==3.2 six==1.11.0 pylibemu==0.5.8 pefile==2017.11.5 yara-python==3.6.3 elasticsearch==5.5.1 esprima==4.0.0.dev12 Flask==0.11.1

Can you please point me to what I am missing?

[buffer]

Apparently your installation failed to copy configuration files to /etc/thug directory. Please take a look at

https://github.com/buffer/thug/blob/master/docker/Dockerfile

for a working Dockerfile.

[vkoppaka]

I needed to not run the apt-get installs in a virtual environment. Once I took virtual environment out, everything worked. Thanks!

[vkoppaka]

My docker instance stopped working without any changes to the Dockerfile (which used to work) with the same error I used to receive "OSError: [Errno 2] No such file or directory: '/etc/thug/plugins'" can you please give me any pointers on how I can debug why /etc/thug/plugins are not getting copied over? I have double checked the Dockerfile to match what is in the repo.

[vkoppaka]

@buffer just curious if there is a way for me to debug why plugins are not getting copied over. Thanks!

[buffer]

I am not really a Docker guru but I would suggest to you to take a look at

https://github.com/buffer/thug/blob/master/docker/Dockerfile

This Dockerfile is working properly. I would use this in you environment to figure out if the issue still happens.