search

yogeshojha / rengine

reNgine is an automated reconnaissance framework for web applications with a focus on highly configurable streamlined recon process via Engines, recon data correlation and organization, continuous monitoring, backed by a database, and simple yet intuitive User Interface. reNgine makes it easy for penetration testers to gather reconnaissance with minimal configuration and with the help of reNgine's correlation, it just makes recon effortless.
https://yogeshojha.github.io/rengine/
GNU General Public License v3.0
7.32k stars 1.1k forks source link

Feature - Add Support For ARM arch #675

Closed ym500 closed 9 months ago

ym500 commented 2 years ago

Is your feature request related to a problem? Please describe. yes, rengine not worked well on arm64 arch due to geckodriver and go installtions they were built for amd

Describe the solution you'd like

just chaning the selenuim and go to support arm

Describe alternatives you've considered Changing the docker file to support arm64 and should work

Docker File 

FROM ubuntu:20.04

ENV GO111MODULE="on"
ENV GOPROXY="https://goproxy.io,direct"
# Labels and Credits
LABEL \
    name="reNgine" \
    author="Yogesh Ojha <yogesh.ojha11@gmail.com>" \
    description="reNgine is a automated pipeline of recon process, useful for information gathering during web application penetration testing."

# Environment Variables
ENV DEBIAN_FRONTEND="noninteractive" \
    DATABASE="postgres"

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Install essentials
RUN apt update -y && apt install -y  --no-install-recommends \
    build-essential \
    cmake \
    firefox \
    gcc \
    git \
    libpq-dev \
    libpq-dev \
    libpcap-dev \
    netcat \
    postgresql \
    python3 \
    python3-dev \
    python3-pip \
    python3-netaddr \
    wget \
    x11-utils \
    xvfb \
    python3-cffi \
    python3-brotli \
    libpango-1.0-0 \
    libpangoft2-1.0-0 \
    geoip-bin \
    geoip-database \
    firefox-geckodriver  #install Download geckodriver

# Download and install go 1.17
RUN wget https://golang.org/dl/go1.17.2.linux-arm64.tar.gz
RUN tar -xvf go1.17.2.linux-arm64.tar.gz
RUN rm go1.17.2.linux-arm64.tar.gz
RUN mv go /usr/local

# ENV for Go
ENV GOROOT="/usr/local/go"
ENV PATH="${PATH}:${GOROOT}/bin"
ENV PATH="${PATH}:${GOPATH}/bin"

ENV GOPATH=$HOME/go
ENV PATH="${PATH}:${GOROOT}/bin:${GOPATH}/bin"

# Make directory for app
WORKDIR /usr/src/app

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Download Go packages
RUN go install -v github.com/hakluke/hakrawler@latest

RUN GO111MODULE=on go install -v -v github.com/bp0lr/gauplus@latest

RUN GO111MODULE=on go install -v github.com/jaeles-project/gospider@latest

RUN go install -v github.com/OWASP/Amass/v3/...@latest

RUN go install -v github.com/ffuf/ffuf@latest

RUN go install -v github.com/tomnomnom/assetfinder@latest
RUN GO111MODULE=on go install -v github.com/tomnomnom/gf@latest
RUN GO111MODULE=on go install -v github.com/tomnomnom/unfurl@latest
RUN GO111MODULE=on go install -v github.com/tomnomnom/waybackurls@latest

RUN GO111MODULE=on go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest
RUN GO111MODULE=on go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest
RUN GO111MODULE=on go install -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest
RUN GO111MODULE=on go install -v github.com/projectdiscovery/naabu/v2/cmd/naabu@latest

# Update Nuclei and Nuclei-Templates
RUN nuclei -update
RUN nuclei -update-templates

# Copy requirements
COPY ./requirements.txt /tmp/requirements.txt
RUN pip3 install --upgrade setuptools pip && \
    pip3 install -r /tmp/requirements.txt

# install eyewitness

RUN python3 -m pip install fuzzywuzzy \
    selenium \
    python-Levenshtein \
    pyvirtualdisplay \
    netaddr

# Copy source code
COPY . /usr/src/app/

# httpx seems to have issue, use alias instead!!!
RUN echo 'alias httpx="/go/bin/httpx"' >> ~/.bashrc
github-actions[bot] commented 2 years ago

👋 Hi @yosefmahmoud, Issues is only for reporting a bug/feature request. Please read documentation before raising an issue https://rengine.wiki For very limited support, questions, and discussions, please join reNgine Discord channel: https://discord.gg/azv6fzhNCE Please include all the requested and relevant information when opening a bug report. Improper reports will be closed without any response.

ved009 commented 1 year ago

The changes work but the only issue is over eyewitness. There is a fix for geckodriver and I might raise a PR for it

psyray commented 9 months ago

Could you try on v2 and reopen issue if problem persists

crazyyyyalex commented 8 months ago

Is your feature request related to a problem? Please describe. yes, rengine not worked well on arm64 arch due to geckodriver and go installtions they were built for amd

Describe the solution you'd like

just chaning the selenuim and go to support arm

Describe alternatives you've considered Changing the docker file to support arm64 and should work

Docker File 

FROM ubuntu:20.04

ENV GO111MODULE="on"
ENV GOPROXY="https://goproxy.io,direct"
# Labels and Credits
LABEL \
    name="reNgine" \
    author="Yogesh Ojha <yogesh.ojha11@gmail.com>" \
    description="reNgine is a automated pipeline of recon process, useful for information gathering during web application penetration testing."

# Environment Variables
ENV DEBIAN_FRONTEND="noninteractive" \
    DATABASE="postgres"

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Install essentials
RUN apt update -y && apt install -y  --no-install-recommends \
    build-essential \
    cmake \
    firefox \
    gcc \
    git \
    libpq-dev \
    libpq-dev \
    libpcap-dev \
    netcat \
    postgresql \
    python3 \
    python3-dev \
    python3-pip \
    python3-netaddr \
    wget \
    x11-utils \
    xvfb \
    python3-cffi \
    python3-brotli \
    libpango-1.0-0 \
    libpangoft2-1.0-0 \
    geoip-bin \
    geoip-database \
    firefox-geckodriver  #install Download geckodriver

# Download and install go 1.17
RUN wget https://golang.org/dl/go1.17.2.linux-arm64.tar.gz
RUN tar -xvf go1.17.2.linux-arm64.tar.gz
RUN rm go1.17.2.linux-arm64.tar.gz
RUN mv go /usr/local

# ENV for Go
ENV GOROOT="/usr/local/go"
ENV PATH="${PATH}:${GOROOT}/bin"
ENV PATH="${PATH}:${GOPATH}/bin"

ENV GOPATH=$HOME/go
ENV PATH="${PATH}:${GOROOT}/bin:${GOPATH}/bin"

# Make directory for app
WORKDIR /usr/src/app

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Download Go packages
RUN go install -v github.com/hakluke/hakrawler@latest

RUN GO111MODULE=on go install -v -v github.com/bp0lr/gauplus@latest

RUN GO111MODULE=on go install -v github.com/jaeles-project/gospider@latest

RUN go install -v github.com/OWASP/Amass/v3/...@latest

RUN go install -v github.com/ffuf/ffuf@latest

RUN go install -v github.com/tomnomnom/assetfinder@latest
RUN GO111MODULE=on go install -v github.com/tomnomnom/gf@latest
RUN GO111MODULE=on go install -v github.com/tomnomnom/unfurl@latest
RUN GO111MODULE=on go install -v github.com/tomnomnom/waybackurls@latest

RUN GO111MODULE=on go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest
RUN GO111MODULE=on go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest
RUN GO111MODULE=on go install -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest
RUN GO111MODULE=on go install -v github.com/projectdiscovery/naabu/v2/cmd/naabu@latest

# Update Nuclei and Nuclei-Templates
RUN nuclei -update
RUN nuclei -update-templates

# Copy requirements
COPY ./requirements.txt /tmp/requirements.txt
RUN pip3 install --upgrade setuptools pip && \
    pip3 install -r /tmp/requirements.txt

# install eyewitness

RUN python3 -m pip install fuzzywuzzy \
    selenium \
    python-Levenshtein \
    pyvirtualdisplay \
    netaddr

# Copy source code
COPY . /usr/src/app/

# httpx seems to have issue, use alias instead!!!
RUN echo 'alias httpx="/go/bin/httpx"' >> ~/.bashrc

had to upgrade Go to 1.20 and chanhe amass link to make it work:

Download and install go 1.20

RUN wget https://go.dev/dl/go1.20.11.linux-arm64.tar.gz

RUN GO111MODULE=on go install -v github.com/owasp-amass/amass/v3/...@latest

AnonymousWP commented 8 months ago

Is your feature request related to a problem? Please describe. yes, rengine not worked well on arm64 arch due to geckodriver and go installtions they were built for amd Describe the solution you'd like just chaning the selenuim and go to support arm Describe alternatives you've considered Changing the docker file to support arm64 and should work Docker File 

FROM ubuntu:20.04

ENV GO111MODULE="on"
ENV GOPROXY="https://goproxy.io,direct"
# Labels and Credits
LABEL \
    name="reNgine" \
    author="Yogesh Ojha <yogesh.ojha11@gmail.com>" \
    description="reNgine is a automated pipeline of recon process, useful for information gathering during web application penetration testing."

# Environment Variables
ENV DEBIAN_FRONTEND="noninteractive" \
    DATABASE="postgres"

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Install essentials
RUN apt update -y && apt install -y  --no-install-recommends \
    build-essential \
    cmake \
    firefox \
    gcc \
    git \
    libpq-dev \
    libpq-dev \
    libpcap-dev \
    netcat \
    postgresql \
    python3 \
    python3-dev \
    python3-pip \
    python3-netaddr \
    wget \
    x11-utils \
    xvfb \
    python3-cffi \
    python3-brotli \
    libpango-1.0-0 \
    libpangoft2-1.0-0 \
    geoip-bin \
    geoip-database \
    firefox-geckodriver  #install Download geckodriver

# Download and install go 1.17
RUN wget https://golang.org/dl/go1.17.2.linux-arm64.tar.gz
RUN tar -xvf go1.17.2.linux-arm64.tar.gz
RUN rm go1.17.2.linux-arm64.tar.gz
RUN mv go /usr/local

# ENV for Go
ENV GOROOT="/usr/local/go"
ENV PATH="${PATH}:${GOROOT}/bin"
ENV PATH="${PATH}:${GOPATH}/bin"

ENV GOPATH=$HOME/go
ENV PATH="${PATH}:${GOROOT}/bin:${GOPATH}/bin"

# Make directory for app
WORKDIR /usr/src/app

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Download Go packages
RUN go install -v github.com/hakluke/hakrawler@latest

RUN GO111MODULE=on go install -v -v github.com/bp0lr/gauplus@latest

RUN GO111MODULE=on go install -v github.com/jaeles-project/gospider@latest

RUN go install -v github.com/OWASP/Amass/v3/...@latest

RUN go install -v github.com/ffuf/ffuf@latest

RUN go install -v github.com/tomnomnom/assetfinder@latest
RUN GO111MODULE=on go install -v github.com/tomnomnom/gf@latest
RUN GO111MODULE=on go install -v github.com/tomnomnom/unfurl@latest
RUN GO111MODULE=on go install -v github.com/tomnomnom/waybackurls@latest

RUN GO111MODULE=on go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest
RUN GO111MODULE=on go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest
RUN GO111MODULE=on go install -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest
RUN GO111MODULE=on go install -v github.com/projectdiscovery/naabu/v2/cmd/naabu@latest

# Update Nuclei and Nuclei-Templates
RUN nuclei -update
RUN nuclei -update-templates

# Copy requirements
COPY ./requirements.txt /tmp/requirements.txt
RUN pip3 install --upgrade setuptools pip && \
    pip3 install -r /tmp/requirements.txt

# install eyewitness

RUN python3 -m pip install fuzzywuzzy \
    selenium \
    python-Levenshtein \
    pyvirtualdisplay \
    netaddr

# Copy source code
COPY . /usr/src/app/

# httpx seems to have issue, use alias instead!!!
RUN echo 'alias httpx="/go/bin/httpx"' >> ~/.bashrc

had to upgrade Go to 1.20 and chanhe amass link to make it work:

FROM ubuntu:20.04

ENV GO111MODULE="on" ENV GOPROXY="https://goproxy.io,direct"

Labels and Credits

LABEL name="reNgine" author="Yogesh Ojha yogesh.ojha11@gmail.com" description="reNgine is a automated pipeline of recon process, useful for information gathering during web application penetration testing."

Environment Variables

ENV DEBIAN_FRONTEND="noninteractive" DATABASE="postgres"

ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1

Install essentials

RUN apt update -y && apt install -y --no-install-recommends build-essential cmake firefox gcc git libpq-dev libpq-dev libpcap-dev netcat postgresql python3 python3-dev python3-pip python3-netaddr wget x11-utils xvfb python3-cffi python3-brotli libpango-1.0-0 libpangoft2-1.0-0 geoip-bin geoip-database firefox-geckodriver #install Download geckodriver

Download and install go 1.20

RUN wget https://go.dev/dl/go1.20.11.linux-arm64.tar.gz RUN tar -xvf go1.20.11.linux-arm64.tar.gz RUN rm go1.20.11.linux-arm64.tar.gz RUN mv go /usr/local

ENV for Go

ENV GOROOT="/usr/local/go" ENV PATH="${PATH}:${GOROOT}/bin" ENV PATH="${PATH}:${GOPATH}/bin"

ENV GOPATH=$HOME/go ENV PATH="${PATH}:${GOROOT}/bin:${GOPATH}/bin"

Make directory for app

WORKDIR /usr/src/app

set environment variables

ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1

Download Go packages

RUN go install -v github.com/hakluke/hakrawler@latest

RUN GO111MODULE=on go install -v -v github.com/bp0lr/gauplus@latest

RUN GO111MODULE=on go install -v github.com/jaeles-project/gospider@latest

RUN GO111MODULE=on go install -v github.com/owasp-amass/amass/v3/...@latest

RUN go install -v github.com/ffuf/ffuf@latest

RUN go install -v github.com/tomnomnom/assetfinder@latest RUN GO111MODULE=on go install -v github.com/tomnomnom/gf@latest RUN GO111MODULE=on go install -v github.com/tomnomnom/unfurl@latest RUN GO111MODULE=on go install -v github.com/tomnomnom/waybackurls@latest

RUN GO111MODULE=on go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest RUN GO111MODULE=on go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest RUN GO111MODULE=on go install -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest RUN GO111MODULE=on go install -v github.com/projectdiscovery/naabu/v2/cmd/naabu@latest

Update Nuclei and Nuclei-Templates

RUN nuclei -update RUN nuclei -update-templates

Copy requirements

COPY ./requirements.txt /tmp/requirements.txt RUN pip3 install --upgrade setuptools pip && pip3 install -r /tmp/requirements.txt

install eyewitness

RUN python3 -m pip install fuzzywuzzy selenium python-Levenshtein pyvirtualdisplay netaddr

Copy source code

COPY . /usr/src/app/

httpx seems to have issue, use alias instead!!!

RUN echo 'alias httpx="/go/bin/httpx"' >> ~/.bashrc

Uhh, what are you trying to do here? Please use proper formatting.