search

CAIDA / pybgpstream

Python bindings for BGPStream
https://bgpstream.caida.org
BSD 2-Clause "Simplified" License
28 stars 22 forks source link

pybgpstream not able to handle HTTP to HTTPS redirections #52

Open dacCR opened 6 months ago

dacCR commented 6 months ago

Hi,

We've been trying to find a way around this for a while now but cannot find one. Searching the Internet has not come up with anything that helps.

Wondering how others are handling this problem of the http links being redirected to https.

dacCR commented 6 months ago

I can't believe I'm the only one experiencing this. Is there a work around? Where should I post to try and get help and simply report this if it's not well known if this is not the place?

salcock commented 6 months ago

Hi there,

Can you provide an example of a situation where a HTTPS redirect fails?

dacCR commented 6 months ago

Hi,

Sure, 

RuntimeError: Could not get next record (is the stream started?)
1704914803 HTTP ERROR: SSL connect error (35)
2024-01-10 19:26:43 111915: bs_transport_file.c:39: ERROR: Could not open http://data.ris.ripe.net/rrc21/2024.01/updates.20240109.1435.gz for reading
2024-01-10 19:26:43 111915: bgpstream_transport.c:97: ERROR: Could not open resource (http://data.ris.ripe.net/rrc21/2024.01/updates.20240109.1435.gz)
2024-01-10 19:26:43 111915: bgpstream_reader.c:169: WARNING: Could not open (http://data.ris.ripe.net/rrc21/2024.01/updates.20240109.1435.gz). Attempt 5 of 5
2024-01-10 19:26:43 111915: bgpstream_reader.c:181: ERROR: Could not open dumpfile (http://data.ris.ripe.net/rrc21/2024.01/updates.20240109.1435.gz) after 5 attempts. Giving up

Based on tests with wget and curl, it's clear that the URLs are being redirected from HTTP to HTTPS. However, while wget automatically follows the redirect and successfully downloads the file, curl does not follow the redirect by default and instead downloads the HTML content of the 301 Moved Permanently page.

In the case of my Python script, it appears that pybgpstream might not be handling the HTTP to HTTPS redirection properly, leading to the SSL connection errors.

hanskuhn commented 6 months ago

I'm just lurking, but glad to see this issue raised.

RouteViews data archive (archive.routeviews.org) does not currently issue a redirect to HTTPS, but it's on my wishlist to implement. I've been hesitating because I had a suspicion that lots of tooling for BGP data will need to be updated. 👀

dacCR commented 6 months ago

I don't know if this helps or not but I decided to try to make sure everything was up to date. It may not be an issue but I'm taking off for several days in a few minutes so will look at this when I get back. However, I thought I would share just in case it's related.

pip freeze > requirements.txt 
pip install -r requirements.txt --upgrade 

root@bgp01:/new# pip freeze > requirements.txt
root@bgp01:/new# pip install -r requirements.txt --upgrade
Requirement already satisfied: attrs==21.2.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 1)) (21.2.0)
Requirement already satisfied: Automat==20.2.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 2)) (20.2.0)
Requirement already satisfied: Babel==2.8.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 3)) (2.8.0)
Requirement already satisfied: bcrypt==3.2.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 4)) (3.2.0)
Requirement already satisfied: blinker==1.4 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 5)) (1.4)
Requirement already satisfied: certifi==2020.6.20 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 6)) (2020.6.20)
Requirement already satisfied: chardet==4.0.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 7)) (4.0.0)
Requirement already satisfied: click==8.0.3 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 8)) (8.0.3)
Requirement already satisfied: cloud-init==23.1.2 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 9)) (23.1.2)
Requirement already satisfied: colorama==0.4.4 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 10)) (0.4.4)
Requirement already satisfied: command-not-found==0.3 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 11)) (0.3)
Requirement already satisfied: configobj==5.0.6 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 12)) (5.0.6)
Requirement already satisfied: constantly==15.1.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 13)) (15.1.0)
Requirement already satisfied: cryptography==3.4.8 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 14)) (3.4.8)
Requirement already satisfied: dbus-python==1.2.18 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 15)) (1.2.18)
Requirement already satisfied: distro==1.7.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 16)) (1.7.0)
Requirement already satisfied: distro-info===1.1build1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 17)) (1.1build1)
Requirement already satisfied: httplib2==0.20.2 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 18)) (0.20.2)
Requirement already satisfied: hyperlink==21.0.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 19)) (21.0.0)
Requirement already satisfied: idna==3.3 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 20)) (3.3)
Requirement already satisfied: importlib-metadata==4.6.4 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 21)) (4.6.4)
Requirement already satisfied: incremental==21.3.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 22)) (21.3.0)
Requirement already satisfied: jeepney==0.7.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 23)) (0.7.1)
Requirement already satisfied: Jinja2==3.0.3 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 24)) (3.0.3)
Requirement already satisfied: jsonpatch==1.32 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 25)) (1.32)
Requirement already satisfied: jsonpointer==2.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 26)) (2.0)
Requirement already satisfied: jsonschema==3.2.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 27)) (3.2.0)
Requirement already satisfied: keyring==23.5.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 28)) (23.5.0)
Requirement already satisfied: launchpadlib==1.10.16 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 29)) (1.10.16)
Requirement already satisfied: lazr.restfulclient==0.14.4 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 30)) (0.14.4)
Requirement already satisfied: lazr.uri==1.0.6 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 31)) (1.0.6)
Requirement already satisfied: MarkupSafe==2.0.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 32)) (2.0.1)
Requirement already satisfied: more-itertools==8.10.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 33)) (8.10.0)
Requirement already satisfied: mysql-connector-python==8.2.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 34)) (8.2.0)
Requirement already satisfied: netifaces==0.11.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 35)) (0.11.0)
Requirement already satisfied: oauthlib==3.2.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 36)) (3.2.0)
Requirement already satisfied: pexpect==4.8.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 37)) (4.8.0)
Requirement already satisfied: protobuf==4.21.12 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 38)) (4.21.12)
Requirement already satisfied: ptyprocess==0.7.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 39)) (0.7.0)
Requirement already satisfied: pyasn1==0.4.8 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 40)) (0.4.8)
Requirement already satisfied: pyasn1-modules==0.2.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 41)) (0.2.1)
Requirement already satisfied: pybgpstream==2.0.2 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 42)) (2.0.2)
Requirement already satisfied: PyGObject==3.42.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 43)) (3.42.1)
Requirement already satisfied: PyHamcrest==2.0.2 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 44)) (2.0.2)
Requirement already satisfied: PyJWT==2.3.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 45)) (2.3.0)
Requirement already satisfied: pyOpenSSL==21.0.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 46)) (21.0.0)
Requirement already satisfied: pyparsing==2.4.7 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 47)) (2.4.7)
Requirement already satisfied: pyrsistent==0.18.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 48)) (0.18.1)
Requirement already satisfied: pyserial==3.5 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 49)) (3.5)
Requirement already satisfied: python-apt==2.4.0+ubuntu1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 50)) (2.4.0+ubuntu1)
Requirement already satisfied: python-dateutil==2.8.2 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 51)) (2.8.2)
Requirement already satisfied: python-debian===0.1.43ubuntu1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 52)) (0.1.43ubuntu1)
Requirement already satisfied: python-magic==0.4.24 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 53)) (0.4.24)
Requirement already satisfied: pytz==2022.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 54)) (2022.1)
Requirement already satisfied: PyYAML==5.4.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 55)) (5.4.1)
Requirement already satisfied: requests==2.25.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 56)) (2.25.1)
Requirement already satisfied: SecretStorage==3.3.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 57)) (3.3.1)
Requirement already satisfied: service-identity==18.1.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 58)) (18.1.0)
Requirement already satisfied: six==1.16.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 59)) (1.16.0)
Requirement already satisfied: sos==4.4 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 60)) (4.4)
Requirement already satisfied: ssh-import-id==5.11 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 61)) (5.11)
Requirement already satisfied: systemd-python==234 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 62)) (234)
Requirement already satisfied: Twisted==22.1.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 63)) (22.1.0)
Requirement already satisfied: ubuntu-advantage-tools==8001 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 64)) (8001)
Requirement already satisfied: ubuntu-drivers-common==0.0.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 65)) (0.0.0)
Requirement already satisfied: ufw==0.36.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 66)) (0.36.1)
Requirement already satisfied: unattended-upgrades==0.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 67)) (0.1)
Requirement already satisfied: urllib3==1.26.5 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 68)) (1.26.5)
Requirement already satisfied: wadllib==1.3.6 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 69)) (1.3.6)
Requirement already satisfied: xkit==0.0.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 70)) (0.0.0)
Requirement already satisfied: zipp==1.0.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 71)) (1.0.0)
Requirement already satisfied: zope.interface==5.4.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 72)) (5.4.0)
Collecting pycairo>=1.16.0
  Downloading pycairo-1.25.1.tar.gz (347 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 347.1/347.1 KB 1.4 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: pycairo
  Building wheel for pycairo (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pycairo (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/cairo
      copying cairo/__init__.py -> build/lib.linux-x86_64-3.10/cairo
      copying cairo/__init__.pyi -> build/lib.linux-x86_64-3.10/cairo
      copying cairo/py.typed -> build/lib.linux-x86_64-3.10/cairo
      running build_ext
      'pkg-config' not found.
      Command ['pkg-config', '--print-errors', '--exists', 'cairo >= 1.15.10']
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pycairo
Failed to build pycairo
ERROR: Could not build wheels for pycairo, which is required to install pyproject.toml-based projects
dacCR commented 5 months ago

Does anyone know how to solve this? Updating everything didn't work.

salcock commented 5 months ago

This is still on my radar, but I have not had a lot of time to investigate the issue so far.

dacCR commented 5 months ago

I'm happy to run more tests for you if it'll help. Just not sure what those could be. Anyone that's built a script and isn't watching closely would not even be aware their code is not working anymore.

salcock commented 5 months ago

Just a quick update -- I believe I've tracked down the problem and have a potential solution.

The underlying problem is within the libbgpstream code. Once I've completed my testing and confirmed that the problem is resolved, I'll submit a PR with the fix.

dacCR commented 5 months ago

Wonderful, thank you for the update.

salcock commented 5 months ago

A PR with the fix has now been submitted and is awaiting review: CAIDA/libbgpstream#245

dacCR commented 5 months ago

I'm not terribly familiar with github, can you let us know when it's approved and/or how to upgrade?

dacCR commented 5 months ago

I just checked the link you shared and it seems no one has reviewed it and it also says merge blocked. I'm a little confused. Why would people use and count on this this if it won't get updated?

dac2020 commented 4 months ago

Upgraded just fine and works, thank you.

essal0815 commented 3 months ago

Hello everyone, I had the same issue. I'm not sure if it is the same root cause for you, but I use MacOS with new M-Series and the pre-installed version of CURL and libcurl are somehow to old and seem to contain this bug. It is actually mentioned in the Install Manual for Mac, but I thought it is only for downloading via curl and so I skipped this step on the first time.

But know I did and after rebuilding anything it worked like a charm. Rough steps for my rebuild. If you build it the first time only see Step 0 and 1 and than continue from the normal install Process:

  1. Earlier step to have /usr/local/bin and /usr/local/include in my build path for pipenv install pybgpstream Hence I am an MacOS for some stupid reason /usr/local is not in the normal Path for clang and co. To make it available I followed this with a slightly modification.

Add to my .bashrc / .zshrc file

export C_INCLUDE_PATH=/opt/homebrew/include:/usr/local/include
export CPLUS_INCLUDE_PATH=/opt/homebrew/include:/usr/local/include
  1. Install curl 
    brew install curl

    Hence it was already installed on the Mac, brew does not automatically links it to the corressponding folders. But when you follow the instructions given by brew it should work

curl is keg-only, which means it was not symlinked into /opt/homebrew,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have curl first in your PATH, run:
  echo 'export PATH="/opt/homebrew/opt/curl/bin:$PATH"' >> ~/.zshrc

For compilers to find curl you may need to set:
  export LDFLAGS="-L/opt/homebrew/opt/curl/lib"
  export CPPFLAGS="-I/opt/homebrew/opt/curl/include"

  1. Rebuild Wandio Since I use Mac, I have to build it from source

    cd /PATH/OF/WANDIO/SRC
make clean
./configure
make
sudo make install

  2. Rebuild libbgpstream Since I use Mac, I have to build it from source

    cd /PATH/OF/LIBBGPSTREAM/SRC
make clean
./configure
make
sudo make install

  3. Reinstall PyBGPStream just to make sure

    pipenv uninstall pybgpstream
pipenv install pybgpstream

Now finally everything works.