Axsgranted commented 10 months ago

Installed the plugin fine, but couldn't get a list of speedtest servers. No error just hangs forever on fetching the list of servers. Decided to swith to Ookla and servers did show up. When I attempted to run a report I got the following:

[31-Aug-2023 13:10:44 America/Phoenix] Phalcon\Mvc\Dispatcher\Exception: OPNsense\Speedtest\UndefinedController handler class cannot be loaded in /usr/local/opnsense/www/index.php:70 Stack trace:

0 [internal function]: Phalcon\Mvc\Dispatcher->throwDispatchException('OPNsense\Speedt...', 2)

1 [internal function]: Phalcon\Dispatcher\AbstractDispatcher->dispatch()

2 /usr/local/opnsense/www/index.php(70): Phalcon\Mvc\Application->handle('/ui/speedtest/u...')

3 {main}

Machine Specs and Version: OPNsense 23.7.3 running on: Dell Optiplex 3050 Intel I5-7600 @ 3.5Ghz (4 Cores) Intel I350-T4 Nic 8G DDR4 256G SSD

mihakralj commented 10 months ago

Check /usr/local/opnsense/scripts/OPNsense/speedtest directory and see if speedtest.csv is in there. While you are in that directory, let's try some basic troubleshooting:

python3 opn_speedtest.py list should generate a list of closest speedtest servers python3 opn_speedtest.py version should list the installed speedtest (in your case Ookla) python3 opn_speedtest.py 0 should run a test on auto-selected server python3 opn_speedtest.py log should list the content of speedtest.csv file (if it exists) python3 opn_speedtest.py stat generates stats from .csv file (if it exists)

Axsgranted commented 10 months ago

Speedtest.csv file is present.

I receive the following trying to run any command {"version": "none", "message": "No speedtest package installed"}

Before you ask, it is installed and I do see the script within the directory. When I try to install the cli via the GUI that's when it hangs on "fetching available Speedtest servers..." and never stops.

Axsgranted commented 10 months ago

Ok did some more testing.

speedtest-cli doesn't work in any scenario If i refresh the page and select Ookla, I can do "socket" testing and I did get a result. However if I attempt to switch to "http speedtest" it complains the module is not installed AND breaks the ookla test. I.e. if I run a test again, ookla test stops immediately and reports nothing.

Axsgranted commented 10 months ago

Anything else I can try?

mimugmail commented 10 months ago

I pushed the update now, please remove and reinstall and try again

Axsgranted commented 10 months ago

Uninstalled and re-installed and was presenting with an instant list of servers to choose from. i did not have to choose to install speedtest-cli or ookla. It just showed list and run test.

If this is by design, then it worked perfectly :)

mihakralj commented 10 months ago

Yes, by design. It will default to whatever is already installed and you can switch the speedtest from Python-based (CLI) to binary from Ookla - and back.

Axsgranted commented 10 months ago

I saw no options to switch in the GUI. Is that a command line option?

mihakralj commented 10 months ago

did you click the Help switch on the Speedtest plugin page top right?

Axsgranted commented 10 months ago

I did not, I see it now ;) "HELP" in general wasn't an intuitive thing for me to think of when changing a config ;) FYI I switched to speedtest HTTP and the same issue occurs. no list if servers.

mihakralj commented 10 months ago

I was young and stupid when I wrote the plugin, didn't really know what I am doing. That was my first adventure in making something for OPNsense and I am surprised that the trash code I created lasted that long and the plugin is still working. Well, I am old and stupid now, but the plugin is opensource on Michael's repo (right here) - and all improvements are very welcome. The change you want needs to be done somewhere in https://github.com/mimugmail/opn-repo/blob/main/net-mgmt/speedtest-community/src/opnsense/mvc/app/views/OPNsense/Speedtest/index.volt

Axsgranted commented 10 months ago

Lol all good. While i'm also no longer young, I haven't touched any real code in many years. I'm afraid I'd do more harm than good ;)

I mean at the end of the day, the sockets test works, and OK with that :)

Axsgranted commented 10 months ago

I could try installing the speedtest-cli manually? I think there are some threads on it already. EDIT: Apparently it's already installed, well at least py39-speedtest-cli-2.1.3 is.

Axsgranted commented 10 months ago

Another update: not sure what happened but after closing the browser and running the previous commands you suggested, when I went back in to the GUI it's working?? however most of the servers are returning "Speedtest server id xxxxx not recognized.", but some DO work.

mihakralj commented 10 months ago

so, as you noticed, there are two speedtest implementations - the python one and the binary one. They can't co-exist as then have the same name; it is the one or the other.

I created a simple bash script install_speedtest.sh (in /usr/local/opnsense/scripts/OPNsense/speedtest) that takes one of three arguments:

sudo ./install_speedtest.sh http - will install speeedtest-cli (python version)
sudo ./install_speedtest.sh socket - will install ookla's speedtest (binary version)
sudo ./install_speedtest.sh delete - will uninstall either/both speedtest packages.

When either package is installed, it has the same name: speedtest with slightly different parameters structure. And, oddly, some server IDs work only under Ookla binary package. My assumption is that old speedtest servers accept both http and socket requests, but new/upgraded speedtest servers accept socket calls only (therefore Ookla binary works in more cases than Python version)

Reloading the Speedtest webpage should cast a new list of closest servers. You can do that manually too:

Ookla binary: speedtest --accept-license --accept-gdpr --servers Python CLI: speedtest --list

in my case, see the difference in listed servers: Ookla:

    ID  Name                           Location             Country
==============================================================================
 51693  Ziply Fiber                    Bothell, WA          United States
 35180  Ziply Fiber                    Seattle, WA          United States
  8864  CenturyLink                    Seattle, WA          United States
 37495  Netprotect                     Seattle, WA          United States
 12192  Nitel                          Seattle, WA          United States
  6199  Wowrack                        Seattle, WA          United States
 49421  Lumen Technologies             Seattle, WA          United States
 22168  Whitesky Communications LLC    Seattle, WA          United States
 50679  Misaka Network, Inc.           Seattle, WA          United States
  1782  Comcast                        Seattle, WA          United States

Python CLI:

35180) Ziply Fiber (Seattle, WA, United States) [2269.78 km]
 8864) CenturyLink (Seattle, WA, United States) [2269.78 km]
49421) Lumen Technologies (Seattle, WA, United States) [2269.78 km]
22168) Whitesky Communications LLC (Seattle, WA, United States) [2269.78 km]
50679) Misaka Network, Inc. (Seattle, WA, United States) [2269.78 km]
 6057) Accretive Networks (Seattle, WA, United States) [2269.78 km]
18532) Wave (Port Orchard, WA, United States) [2288.81 km]
54406) Whidbey Telecom (Langley, WA, United States) [2291.76 km]
40251) iFiber Communications (Shelton, WA, United States) [2310.44 km]
31110) City of Anacortes (Anacortes, WA, United States) [2325.07 km]

mihakralj commented 10 months ago

Apparently it's already installed, well at least py39-speedtest-cli-2.1.3 is

The nightmare of slippery names is a problem with both packages - Python and Binary. Therefore I updated the install_speedtest.sh to try to catch the name of the latest available package:

for Python package (which is in OPNSense repo): PYTHON_PKG_NAME=$(pkg info -x 'python3*' | grep 'python3' | sed -n 's/^python$[0-9]*$.*/py\1-speedtest-cli/p')
for Binary version (not in any standard repos, need direct URL): OOKLA_URL=$(curl -s "https://www.speedtest.net/apps/cli" | xmllint --html --xpath "string(//*[@id='freebsd']/pre/code[7]/text())" - 2>/dev/null | awk -F'"' '{print $2}') (you can echo $PYTHON_PKG_NAME and echo $OOKLA_URL to see the calculated current package name/url)

Axsgranted commented 10 months ago

Apparently it's already installed, well at least py39-speedtest-cli-2.1.3 is

The nightmare of slippery names is a problem with both packages - Python and Binary. Therefore I updated the install_speedtest.sh to try to catch the name of the latest available package:

for Python package (which is in OPNSense repo): PYTHON_PKG_NAME=$(pkg info -x 'python3*' | grep 'python3' | sed -n 's/^python$[0-9]*$.*/py\1-speedtest-cli/p')

for Binary version (not in any standard repos, need direct URL): OOKLA_URL=$(curl -s "https://www.speedtest.net/apps/cli" | xmllint --html --xpath "string(//*[@id='freebsd']/pre/code[7]/text())" - 2>/dev/null | awk -F'"' '{print $2}') (you can echo $PYTHON_PKG_NAME and echo $OOKLA_URL to see the calculated current package name/url)

Yup I was actually just checking this but I oddly had to change execute permissions to get it to run, which begs the question, without looking at the repo, how does this run in the first place? do you copy, run, delete or something? FYI the package name and URL both enumerated perfectly.

Also curious, if new/updated servers are all socket only why not just have the one binary option going forward and simplify?

mihakralj commented 10 months ago

@mimugmail do we have the exec flag set in the latest speedtest package? I cannot see the settings in the Github source tree - we should check that there is a chmod +x on all .sh commands in the package.

mihakralj commented 10 months ago

if new/updated servers are all socket only why not just have the one binary option going forward and simplify?

Ookla binary is not open source and Python CLI is. OPNsense prefers OSS whenever possible. So, default is OSS speedtest with an option to replace it with a (proprietary) binary.

mimugmail commented 10 months ago

You can add a PR with a Post install file and chmod?

mihakralj commented 9 months ago

I tested around and packager (pkg create) honors flags on the file as it packages it into the archive; we just need to set the flag right in the repo; will submit the PR for it.

BTW, I found out that I can package the .txz (or .pkg) for FreeBSD on Linux as well, using bsdtar:

install bsdtar on Linux: apt install bsdtar (Linux uses GNU tar by default)
package with bsdtar -cJf <package.tzx> -s'|^\./|/|' -P +MANIFEST +COMPACT_MANIFEST ./<plugin dev path>

The -s is a substitution option that will remove ./ at the beginning of filenames and make paths absolute, allowing archive to be extracted to the absolute path.

mimugmail commented 9 months ago

Why not create a file like here and adding a chmod

mihakralj commented 9 months ago

that works too, but it is unnecessary if we set the +x flag on the file before packing it. Let me play with it.

mihakralj commented 9 months ago

https://github.com/mimugmail/opn-repo/pull/180 has +x flag set on .sh and .py files required for this plugin. Let's repackage and see if FreeBSD documentation is right...

mihakralj commented 9 months ago

this issue is ready to be closed.

mimugmail / opn-repo

Speedtest not working at all #177

0 [internal function]: Phalcon\Mvc\Dispatcher->throwDispatchException('OPNsense\Speedt...', 2)

1 [internal function]: Phalcon\Dispatcher\AbstractDispatcher->dispatch()

2 /usr/local/opnsense/www/index.php(70): Phalcon\Mvc\Application->handle('/ui/speedtest/u...')

3 {main}