Feature request: make raw data available for download

[cracket]

Would it be possible to upload raw data from tests into some network location available for download?

I'd like to do some statistics for older boards and hardware configuration, and current pibenchmarks.com webpage is very nice, but it's tough to find more complex trends and stats based on top 15 or last 15 records retrieved and focused on RPi 4B.

[TheRemote]

I definitely think this would be possible. It wouldn't be anything like a giant raw data dump but it's basically an API. It kind of already works this way but it's completely invisible to you guys.

Observe:

james@jamesgigabyte-linux:~/Desktop$ curl -k -L https://pibenchmarks.com/api/benchmark/62444 | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10751    0 10751    0     0  56385      0 --:--:-- --:--:-- --:--:-- 56584
{
  "result": [
    {
      "submission_id": 62444,
      "score": 7279,
      "useralias": "Anonymous",
      "brand": "Inland Professional",
      "model": "SATASSD",
      "manufacturer": "Inland",
      "class": "SSD (2.5\" SATA)",
      "capacity": "256G",
      "product": "SSD",
      "submission_date": "2022-09-05 11:08:05",
      "note": "Inland Pro 256GB on Pi4 via USB3 using Sabrent EC-OCUB with UASP enabled",
      "sdtype": null,
      "sdspec": null,
      "data_sd_mid": null,
      "data_sd_oid": null,
      "data_sd_speedclass": null,
      "data_sd_appclass": null,
      "data_sd_uhsclass": null,
      "data_sd_videoclass": null,
      "revision": null,
      "datemanufactured": "",
      "hostmodel": "Raspberry Pi 4 Model B Rev 1.2",
      "hostmanufacturer": "Raspberry Pi Foundation",
      "hostos": "Raspbian GNU/Linux 11 (bullseye)",
      "hostarchitecture": "armv7l",
      "hostcpuclock": 1500,
      "hostcoreclock": 500,
      "hostramclock": 1600,
      "hostsdclock": 0,
      "ddwritespeed": 162,
      "hdparmdisk": 264.74,
      "hdparmcached": 263.81,
      "fio4krandwriteiops": 7658,
      "fio4krandreadiops": 17268,
      "fio4krandwritespeed": 30635,
      "fio4krandreadspeed": 69072,
      "io4kwrite": 24006,
      "io4kread": 24808,
      "io4krandread": 15976,
      "io4krandwrite": 30367,
      "image": "Inland_SSD_Professional.jpg",
      "data_hdd_rpm": "Solid State Device",
      "data_interface": "SATA",
      "data_formfactor": "2.5 inch",
      "data_adapter_usbid": "152d:0576",
      "data_adapter_driver": "uas",
      "data_adapter_idvendor": "152d",
      "data_adapter_idproduct": "0576",
      "popularity": 118,
      "ranking": 433,
      "countbrand": 64,
      "avgscore": 6008,
      "rank_product": 426,
      "rank_class": 158,
      "offer_url": "https://www.amazon.com/dp/B07BD32RLK?tag=codemallet-20&linkCode=ogi&th=1&psc=1",
      "scoredifference": 1271,
      "totalbrands": 1314,
      "totalclasses": 351,
      "totalproducts": 760,
      "data_usb_vendor": "JMicron Technology Corp. / JMicron USA Technology Corp.",
      "data_usb_model": "",
      "usb_version": "3.10",
      "usb_speed": "5000",
      "inxi_cpu_model": "ARMv7 v7l",
      "inxi_cpu_cores": "4",
      "inxi_mobo_vendor": null,
      "inxi_mobo_model": null,
      "inxi_ram_total": "4 GB",
      "data_cpu_oc": 0,
      "data_core_oc": 0,
      "data_ram_oc": 0,
      "data_sd_oc": -50,
      "test_inxi": "System:\n  Kernel: 5.15.61-v7l+ arch: armv7l bits: 32 compiler: N/A parameters: coherent_pool=1M\n    8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1\n    video=HDMI-A-1:1920x1080M@60 smsc95xx.macaddr=XX:XX:XX:XX:XX:XX vc_mem.mem_base=0x3eb00000\n    vc_mem.mem_size=0x3ff00000 console=ttyS0,115200 console=tty1 root=PARTUUID=2bae8b50-02\n    rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles\n  Console: tty 1 wm: Mutter v: 3.38.6 DM: LightDM v: 1.26.0 Distro: Raspbian GNU/Linux 11\n    (bullseye)\nMachine:\n  Type: ARM System: Raspberry Pi 4 Model B Rev 1.2 details: BCM2711 rev: c03112 Battery:\n  Device-1: hidpp_battery_0 model: Logitech Wireless Touch Keyboard K400 Plus charge: 55% (should be ignored) rechargeable: yes status: discharging\nMemory:\n  RAM: total: 3.82 GiB used: 909.9 MiB (23.2%) gpu: 76 MiB\n  RAM Report: smbios: No SMBIOS data for dmidecode to process\nPCI Slots:\n  Message: No ARM data found for this feature.\nCPU:\n  Info: model: ARMv7 v7l variant: cortex-a72 bits: 32 type: MCP arch: v7l family: 7 model-id: 0\n    stepping: 3\n  Topology: cpus: 1x cores: 4 smt: N/A cache: N/A\n  Speed (MHz): avg: 800 min/max: 600/1500 scaling: driver: cpufreq-dt governor: ondemand cores:\n    1: 800 2: 800 3: 800 4: 800 bogomips: 1080\n  Features: crc32 edsp evtstrm fastmult half idiva idivt lpae neon thumb tls vfp vfpd32 vfpv3\n    vfpv4\n  Vulnerabilities:\n  Type: itlb_multihit status: Not affected\n  Type: l1tf status: Not affected\n  Type: mds status: Not affected\n  Type: meltdown status: Not affected\n  Type: mmio_stale_data status: Not affected\n  Type: retbleed status: Not affected\n  Type: spec_store_bypass status: Not affected\n  Type: spectre_v1 mitigation: __user pointer sanitization\n  Type: spectre_v2 status: Vulnerable\n  Type: srbds status: Not affected\n  Type: tsx_async_abort status: Not affected\nGraphics:\n  Device-1: bcm2711-hdmi0 driver: vc4_hdmi v: N/A bus-ID: N/A chip-ID: brcm:fef00700\n    class-ID: hdmi\n  Device-2: bcm2711-hdmi1 driver: vc4_hdmi v: N/A bus-ID: N/A chip-ID: brcm:fef05700\n    class-ID: hdmi\n  Device-3: bcm2711-vc5 driver: vc4_drm v: N/A bus-ID: N/A chip-ID: brcm:gpu class-ID: gpu\n  Display: x11 server: X.Org v: 1.20.11 with: Xwayland compositor: Mutter v: 3.38.6 driver: X:\n    loaded: modesetting unloaded: fbdev\n    gpu: vc4-drm,vc4_crtc,vc4_dpi,vc4_dsi,vc4_firmware_kms,vc4_hdmi,vc4_hvs,vc4_txp,vc4_v3d,vc4_vec\n    display-ID: :0 screens: 1\n  Screen-1: 0 s-res: 1920x1080 s-dpi: 96 s-size: 508x285mm (20.00x11.22\") s-diag: 582mm (22.93\")\n  Monitor-1: HDMI-A-2 mapped: HDMI-2 model: MSI MAG241C built: 2019\n    res: 1920x1080 dpi: 94 gamma: 1.2 chroma: red: x: 0.678 y: 0.314 green: x: 0.286 y: 0.651 blue:\n    x: 0.149 y: 0.075 white: x: 0.314 y: 0.329 size: 521x293mm (20.51x11.54\") diag: 598mm (23.5\")\n    ratio: 16:9 modes: 1920x1080, 1680x1050, 1280x1024, 1440x900, 1280x960, 1152x864, 1280x720,\n    1440x576, 1024x768, 1440x480, 800x600, 720x576, 720x480, 640x480, 720x400\n  EDID-Warnings: 1: parse_edid: unhandled CEA mode 75 2: parse_edid: unhandled CEA mode 76\n  Message: Unable to show GL data. Required tool glxinfo missing.\nAudio:\n  Device-1: bcm2711-hdmi0 driver: vc4_hdmi bus-ID: N/A chip-ID: brcm:fef00700 class-ID: hdmi\n  Device-2: bcm2711-hdmi1 driver: vc4_hdmi bus-ID: N/A chip-ID: brcm:fef05700 class-ID: hdmi\n  Device-3: bcm2835-audio driver: bcm2835_audio bus-ID: N/A chip-ID: brcm:bcm2835_audio\n    class-ID: bcm2835_audio\n  Sound Server-1: ALSA v: k5.15.61-v7l+ running: yes\n  Sound Server-2: PulseAudio v: 14.2 running: yes\n  Sound Server-3: PipeWire v: 0.3.19 running: yes\nNetwork:\n  Device-1: bcm2835-mmc driver: mmc_bcm2835 v: N/A port: N/A bus-ID: N/A chip-ID: brcm:fe300000\n    class-ID: mmcnr\n  IF: wlan0 state: down mac: \n  Device-2: bcm2711-genet-v5 driver: bcmgenet v: N/A port: N/A bus-ID: N/A\n    chip-ID: brcm:fd580000 class-ID: ethernet\n  IF: eth0 state: up speed: 1000 Mbps duplex: full mac: \n  IP v4:  type: noprefixroute scope: global broadcast: \n  IP v6:  type: dynamic mngtmpaddr scope: global\n  IP v6:  scope: link\n  WAN IP: \nBluetooth:\n  Device-1: pl011 driver: uart_pl011 bus-ID: N/A chip-ID: arm:fe201000 class-ID: serial\n  Report: hciconfig ID: hci0 rfk-id: 1 state: down bt-service: enabled,running rfk-block:\n    hardware: no software: yes address: \n  Info: acl-mtu: 1021:8 sco-mtu: 64:1 link-policy: rswitch sniff link-mode: slave accept\nLogical:\n  Message: No logical block device data found.\nRAID:\n  Message: No RAID data found.\nDrives:\n  Local Storage: total: 238.47 GiB used: 88.58 GiB (37.1%)\n  ID-1: /dev/sda maj-min: 8:0 type: USB model: SATA SSD size: 238.47 GiB block-size:\n    physical: 4096 B logical: 512 B type: SSD rev: 1214 scheme: MBR\n  SMART Message: Unknown USB bridge. Flash drive/Unsupported enclosure?\n  Message: No optical or floppy data found.\nPartition:\n  ID-1: / raw-size: 238.22 GiB size: 233.42 GiB (97.99%) used: 88.53 GiB (37.9%) fs: ext4\n    block-size: 4096 B dev: /dev/sda2 maj-min: 8:2 label: rootfs\n    uuid: b101bb80-3338-4b94-a775-b3844f8f2aa8\n  ID-2: /boot raw-size: 256 MiB size: 252 MiB (98.46%) used: 49.5 MiB (19.6%) fs: vfat\n    block-size: 512 B dev: /dev/sda1 maj-min: 8:1 label: boot uuid: 17B6-FC00\nSwap:\n  Kernel: swappiness: 60 (default) cache-pressure: 100 (default)\n  ID-1: swap-1 type: file size: 100 MiB used: 0 KiB (0.0%) priority: -2 file: /var/swap\nUnmounted:\n  Message: No unmounted partitions found.\nUSB:\n  Hub-1: 1-0:1 info: Hi-speed hub with single TT ports: 1 rev: 2.0 speed: 480 Mb/s\n    chip-ID: 1d6b:0002 class-ID: 0900\n  Hub-2: 1-1:2 info: VIA Labs Hub ports: 4 rev: 2.1 speed: 480 Mb/s power: 100mA\n    chip-ID: 2109:3431 class-ID: 0900\n  Device-1: 1-1.3:3 info: Holtek USB Gaming Mouse type: Mouse,Keyboard,HID\n    driver: hid-generic,usbhid interfaces: 3 rev: 2.0 speed: 12 Mb/s power: 100mA chip-ID: 04d9:fc38\n    class-ID: 0300\n  Device-2: 1-1.4:4 info: Logitech Unifying Receiver type: Keyboard,Mouse,HID\n    driver: logitech-djreceiver,usbhid interfaces: 3 rev: 2.0 speed: 12 Mb/s power: 98mA\n    chip-ID: 046d:c52b class-ID: 0300\n  Hub-3: 2-0:1 info: Super-speed hub ports: 4 rev: 3.0 speed: 5 Gb/s chip-ID: 1d6b:0003\n    class-ID: 0900\n  Device-1: 2-2:2 info: JMicron / USA Gen1 SATA 6Gb/s Bridge type: Mass Storage driver: uas\n    interfaces: 1 rev: 3.1 speed: 5 Gb/s power: 896mA chip-ID: 152d:0576 class-ID: 0806\n    Sensors:\n  Missing: Required tool sensors not installed. Check --recommends\nRepos:\n  Packages: pm: dpkg pkgs: 1729 libs: 1008 tools: apt,apt-get\n  Active apt repos in: /etc/apt/sources.list\n    1: deb http: //raspbian.raspberrypi.org/raspbian/ bullseye main contrib non-free rpi\n  Active apt repos in: /etc/apt/sources.list.d/box86.list\n    1: deb https: //itai-nelken.github.io/weekly-box86-debs/debian/ /\n  Active apt repos in: /etc/apt/sources.list.d/raspi.list\n    1: deb http: //archive.raspberrypi.org/debian/ bullseye main\nProcesses:\n  CPU top: 5 of 219\n  1: cpu: 17.8% command: chromium-browser-v7 pid: 1184 mem: 179.6 MiB (4.6%)\n  2: cpu: 12.1% command: chromium-browser-v7 pid: 1255 mem: 107.2 MiB (2.7%)\n  3: cpu: 11.4% command: chromium-browser-v7 pid: 1335 mem: 210.1 MiB (5.4%)\n  4: cpu: 6.5% command: packagekitd pid: 1626 mem: 18.8 MiB (0.4%)\n  5: cpu: 5.0% command: xorg pid: 656 mem: 76.0 MiB (1.9%)\n  Memory top: 5 of 219\n  1: mem: 210.1 MiB (5.4%) command: chromium-browser-v7 pid: 1335 cpu: 11.4%\n  2: mem: 179.6 MiB (4.6%) command: chromium-browser-v7 pid: 1184 cpu: 17.8%\n  3: mem: 109.4 MiB (2.8%) command: chromium-browser-v7 pid: 1454 cpu: 4.1%\n  4: mem: 108.4 MiB (2.8%) command: chromium-browser-v7 pid: 1370 cpu: 4.0%\n  5: mem: 107.2 MiB (2.7%) command: chromium-browser-v7 pid: 1255 cpu: 12.1%\nInfo:\n  Processes: 219 Uptime: 3m Init: systemd v: 247 target: graphical (5) default: graphical\n  tool: systemctl Compilers: gcc: 10.2.1 alt: 10 Client: shell wrapper v: 5.1.4-release\n  inxi: 3.3.21\n",
      "amz_us": "https://www.amazon.com/dp/B07BD32RLK?tag=codemallet-20&linkCode=ogi&th=1&psc=1",
      "amz_ca": "https://amzn.to/3daRjza",
      "amz_uk": "https://amzn.to/3wZG5po",
      "amz_jp": "https://amzn.to/3wXnTg3",
      "amz_au": "https://amzn.to/3h2ssPs",
      "amz_de": "https://amzn.to/3qrAFkd",
      "amz_es": "https://amzn.to/3wXgSMe",
      "amz_fr": "https://amzn.to/3wZwCOH",
      "amz_it": "https://amzn.to/3vZSxDX",
      "amz_nl": "https://amzn.to/3wXgUDQ",
      "amz_pl": "https://amzn.to/3gWuQbm",
      "amz_se": "https://amzn.to/3gVWl5e",
      "amz_sg": "https://amzn.to/3wVeuFJ",
      "ali": "https://s.click.aliexpress.com/e/_AtYh9V",
      "host_amz_us": "https://amzn.to/3EuPz0k",
      "host_ali": "https://s.click.aliexpress.com/e/_ANOf9z",
      "hostbrand": "Raspberry Pi 4B 1.2"
    }
  ],
  "response_code": 200,
  "response_desc": "OK"
}

That is for an individual benchmark. Let's say you wanted to do something like top 15 records for the 4B. If you go to the Pi 4B benchmark page on Pi Benchmarks you can get what we need for the URL:

james@jamesgigabyte-linux:~/Desktop$ curl -k -L https://pibenchmarks.com/api/board/Raspberry_Pi_4_Model_B_Rev_1.2 | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6462    0  6462    0     0  28562      0 --:--:-- --:--:-- --:--:-- 28466
{
  "result": [
    {
      "id": 409,
      "manufacturer": "Raspberry Pi Foundation",
      "model": "Raspberry Pi 4 Model B Rev 1.2",
      "countboard": 6979,
      "popularity": 3,
      "hits": 22,
      "fastest": 11337,
      "fastestid": 25281,
      "slowest": 105,
      "slowestid": 59765,
      "avgscore": 4304,
      "ranking": 357,
      "image": "Pi_4ModelB.jpg",
      "brand": "Raspberry Pi 4B 1.2",
      "asin": "B07TC2BK1X",
      "price": 159.49,
      "price_updated": "2022-09-04 16:53:17",
      "offer_url": "https://www.amazon.com/dp/B07TC2BK1X?tag=codemallet-20&linkCode=ogi&th=1&psc=1",
      "sellcondition": "New",
      "upc": "765756931182",
      "freeshipping": 1,
      "ali": "https://s.click.aliexpress.com/e/_ANOf9z",
      "amz_us": "https://amzn.to/3EuPz0k",
      "amz_ca": "https://amzn.to/3v1Ws6h",
      "amz_uk": "https://amzn.to/3rE7vAi",
      "amz_jp": "https://amzn.to/3Os3AjO",
      "amz_au": "https://amzn.to/3Mmy2tJ",
      "amz_de": "https://amzn.to/3MlZRlU",
      "amz_es": "https://amzn.to/3Oq8BcU",
      "amz_fr": "https://amzn.to/3EzloF2",
      "amz_it": "https://amzn.to/3xHCaki",
      "amz_nl": "https://amzn.to/36A9cYk",
      "amz_pl": "https://amzn.to/3vvPncX",
      "amz_se": "https://amzn.to/3MhdeE6",
      "amz_sg": "https://amzn.to/3vtFDA9",
      "top": [
        {
          "submission_id": 25281,
          "score": 11337,
          "useralias": "Lou",
          "submission_date": "2020-05-23 19:49:54",
          "note": "Sandisk Portable SSD Extreme Pro",
          "model": "ExtremeSSD",
          "ranking": 0
        },
        {
          "submission_id": 10192,
          "score": 10952,
          "useralias": "Anonymous",
          "submission_date": "2020-02-23 17:53:39",
          "note": "Samsung 860 EVO Orico Transparent USB 3.1 Pi 4",
          "model": "SSD860EVO250GB",
          "ranking": 1
        },
        {
          "submission_id": 28154,
          "score": 10932,
          "useralias": "Anonymous",
          "submission_date": "2020-06-27 10:02:16",
          "note": "Samsung 860 with Orico usb 3.0",
          "model": "SSD860EVO250GB",
          "ranking": 2
        },
        {
          "submission_id": 23471,
          "score": 10855,
          "useralias": "dasza77",
          "submission_date": "2020-05-07 08:00:43",
          "note": "",
          "model": "CT500MX500SSD1",
          "ranking": 3
        },
        {
          "submission_id": 25372,
          "score": 10663,
          "useralias": "karaiskos",
          "submission_date": "2020-05-24 16:49:47",
          "note": "",
          "model": "SSD970EVOPlus",
          "ranking": 4
        },
        {
          "submission_id": 25626,
          "score": 10619,
          "useralias": "Anonymous",
          "submission_date": "2020-05-26 20:14:39",
          "note": "SSD Extreme Overclock 2100",
          "model": "ExtremePortableSSD",
          "ranking": 5
        },
        {
          "submission_id": 24133,
          "score": 10543,
          "useralias": "Anonymous",
          "submission_date": "2020-05-12 13:18:50",
          "note": "Samsung T5 Portable - overclock",
          "model": "PortableSSDT5",
          "ranking": 6
        },
        {
          "submission_id": 25277,
          "score": 10500,
          "useralias": "karaiskos",
          "submission_date": "2020-05-23 18:52:31",
          "note": "",
          "model": "SSD970EVOPlus",
          "ranking": 7
        },
        {
          "submission_id": 26966,
          "score": 10458,
          "useralias": "PI4_4 Bob",
          "submission_date": "2020-06-13 05:27:17",
          "note": "KingSpec Z3 SSD om pi4 USB",
          "model": "Z3-128",
          "ranking": 8
        },
        {
          "submission_id": 25225,
          "score": 10402,
          "useralias": "Anonymous",
          "submission_date": "2020-05-23 01:11:23",
          "note": "Samsung SSD 210",
          "model": "PortableSSDT5",
          "ranking": 9
        },
        {
          "submission_id": 13789,
          "score": 10330,
          "useralias": "Anonymous",
          "submission_date": "2020-03-26 14:12:56",
          "note": "M2 SSD",
          "model": "TS120GMTS420S",
          "ranking": 10
        },
        {
          "submission_id": 19959,
          "score": 10312,
          "useralias": "pumatlarge",
          "submission_date": "2020-04-23 02:58:47",
          "note": "Pioneer M.2 APS-SE20G 512GB SSD on Pi 4 using UGreen M.2 Portable HD Enclosure",
          "model": "APS-SE20G-512",
          "ranking": 11
        },
        {
          "submission_id": 23419,
          "score": 10306,
          "useralias": "FireTaz",
          "submission_date": "2020-05-06 18:00:31",
          "note": "x825 BX500",
          "model": "CT1000BX500SSD1",
          "ranking": 12
        },
        {
          "submission_id": 19683,
          "score": 10303,
          "useralias": "pumatlarge",
          "submission_date": "2020-04-21 06:47:05",
          "note": "Pioneer APS-SE20G 512GB M.2 SSD on Pi 4 using UGreen M.2 Portable HD Enclosure",
          "model": "APS-SE20G-512",
          "ranking": 13
        },
        {
          "submission_id": 28443,
          "score": 10298,
          "useralias": "Gibbet",
          "submission_date": "2020-07-02 06:04:58",
          "note": "Raspian Pi4b 4GB Samsung T5 portable SSD",
          "model": "PortableSSDT5",
          "ranking": 14
        }
      ],
      "recent": [
        {
          "submission_id": 62444,
          "score": 7279,
          "useralias": "Anonymous",
          "submission_date": "2022-09-05 11:08:05",
          "note": "Inland Pro 256GB on Pi4 via USB3 using Sabrent EC-OCUB with UASP enabled",
          "model": "SATASSD",
          "ranking": 0
        },
        {
          "submission_id": 62443,
          "score": 6032,
          "useralias": "Anonymous",
          "submission_date": "2022-09-05 09:10:46",
          "note": "Inland Professional 256GB on Pi4 4GB using Inland GDST01 adapter via USB3 with UASP enabled",
          "model": "SATASSD",
          "ranking": 1
        },
        {
          "submission_id": 62439,
          "score": 4647,
          "useralias": "Anonymous",
          "submission_date": "2022-09-05 09:04:11",
          "note": "Inland Professional 256GB SSD on Pi4 4GB using Inland GDST01 adapter USB3 with UASP disabled",
          "model": "SATASSD",
          "ranking": 2
        },
        {
          "submission_id": 62438,
          "score": 5111,
          "useralias": "Anonymous",
          "submission_date": "2022-09-05 06:21:32",
          "note": "Inland Professional 256GB SSD on Pi4 4GB using Sabrent EC-UASP enclosure on USB3 port (quirk cmd edit disable UASP)",
          "model": "SATASSD",
          "ranking": 3
        },
        {
          "submission_id": 62436,
          "score": 5304,
          "useralias": "Anonymous",
          "submission_date": "2022-09-05 05:54:36",
          "note": "Inland Professional 256GB SSD on Pi4 4GB using Sabrent EC-UASP enclosure on USB3 port",
          "model": "SATASSD",
          "ranking": 4
        },
        {
          "submission_id": 62435,
          "score": 2432,
          "useralias": "Anonymous",
          "submission_date": "2022-09-05 05:23:20",
          "note": "Inland Professional 256GB on Pi4 4GB using Sabrent EC-UASP enclosure",
          "model": "SATASSD",
          "ranking": 5
        },
        {
          "submission_id": 62412,
          "score": 7114,
          "useralias": "Anonymous",
          "submission_date": "2022-09-03 22:12:15",
          "note": "pi4",
          "model": "64GB",
          "ranking": 6
        },
        {
          "submission_id": 62407,
          "score": 7849,
          "useralias": "Anonymous",
          "submission_date": "2022-09-03 19:32:44",
          "note": "ssd pi4",
          "model": "CS900240GBSSD",
          "ranking": 7
        },
        {
          "submission_id": 62395,
          "score": 1100,
          "useralias": "Anonymous",
          "submission_date": "2022-09-03 11:40:56",
          "note": "test",
          "model": "SC32G",
          "ranking": 8
        },
        {
          "submission_id": 62365,
          "score": 1161,
          "useralias": "Anonymous",
          "submission_date": "2022-09-02 03:11:41",
          "note": "",
          "model": "EB1QT",
          "ranking": 9
        },
        {
          "submission_id": 62358,
          "score": 4840,
          "useralias": "pi",
          "submission_date": "2022-09-01 13:53:20",
          "note": "pi",
          "model": "SSD850EVO250GB",
          "ranking": 10
        },
        {
          "submission_id": 62357,
          "score": 4827,
          "useralias": "GzOnE",
          "submission_date": "2022-09-01 13:51:00",
          "note": "",
          "model": "SSD850EVO250GB",
          "ranking": 11
        },
        {
          "submission_id": 62339,
          "score": 7801,
          "useralias": "Anonymous",
          "submission_date": "2022-09-01 03:35:19",
          "note": "ssdm2usb3",
          "model": "SSKStorage",
          "ranking": 12
        },
        {
          "submission_id": 62315,
          "score": 1116,
          "useralias": "Anonymous",
          "submission_date": "2022-08-31 02:59:07",
          "note": "schijf",
          "model": "SB32G",
          "ranking": 13
        },
        {
          "submission_id": 62309,
          "score": 2282,
          "useralias": "Kingston",
          "submission_date": "2022-08-30 14:47:51",
          "note": "",
          "model": "SD64G",
          "ranking": 14
        }
      ],
      "totalboards": 537
    }
  ],
  "response_code": 200,
  "response_desc": "OK"
}

Okay, so with those tools what you want to do should be trivial. The Pi 4Bs are broken up into revisions to try to capture any differences/changes between different Pi 4B versions.

That means you would program your bot / scraper to scrape all revisions of the Pi 4B and then you could calculate the numbers from there. It's in JSON already which makes it super easy to work with programmatically.

Can you take a look at those and see if it would let you do what you're trying to do here (and if not what it is missing as I could flesh out the API if needed). If it works for you I could happily document the API for sure for everyone to use (which sounds almost silly to say now that I've shown how powerful it is, of course I should document it). At any rate I think this should get you close to what you're trying to do here!

Bonus tip: You can basically put "/api/" in front of any URL on pibenchmarks.com including the search feature. This will also give you some additional programmatic options if you are looking for certain things. Others I may need to add in if there's nothing like what you're trying to do yet (I'd be willing to do this though).

The API is probably powerful enough for people to create a complete replacement to pibenchmarks.com either as a web site or as an app. If someone wants to give that a go be my guest as I literally made that site to learn how to write a web site from scratch in PHP so I am by no means a visual web design expert. My backend is pretty nice though!

[cracket]

Thank you - this solution is fine for me. I wasn't just sure how your web page hosting behaves when some ip shoots 60k requests to get each benchmark report json separately. The rest is fairly easy - transform json to flat files, group by sbc and correlate with SD reader features in each one.

[TheRemote]

Hey cracket,

Excellent, thanks for asking this, take care!

[TheRemote]

Oh, I should probably have addressed the 60k requests. Right now it's running on an extremely powerful instance. My main web site (with a lot more traffic) is a $10/mo Lightsail instance and the PiBenchmarks one is a $40 instance in order to have a large amount of RAM (and additional CPU cores). It should be able to take it (I pay way more than enough to have the power for it mostly to get the RAM but the CPU cores should help with any load as well).

There are actually about 36,006 benchmarks right now. The reason the counter is up to 60k+ is because there was an attack somewhere in the middle where someone submitted 20-25k fake benchmarks (which were all deleted). Because some people had submitted legitimate benchmarks during the attack (and what would have been afterward) I ended up leaving the counter alone vs. resetting them and fudging ids of the other benchmarks to preserve the integrity of the real data. You will only really have to scrape about 36k at time of writing to get all of them.

You could probably even program it to skip this "gap" for even trying to scrape them (or maybe I should make a bulk API function meant to do this perhaps and return a list of indexes maybe of active benchmarks). If you want to program it to just discard the null results though this will work without causing any difficulty and I'm not worried about load on the server at this time.

As long as it's just people within the community using it I don't think it will be a problem if you want to go totally crazy with it. If it somehow started being abused or the activity was somehow high enough to impact the server or it's stability (maybe by inappropriate programming of something that became really popular that wasn't caching, who knows) then presumably I would have to make some kind of API key system to prevent abuse.

In my eyes that would be a good problem to have though as it means the data is valuable and interesting enough that people are trying to abuse it or whatever happens that forces some kind of restricted API key in the future but for now I would say to go totally crazy with it and try to do whatever you want.

If you run into any limitations or are able to cause some kind of problem through your testing/interactions that would be helpful to me overall as you have a track record of being willing to report issues and interact with me so I would be more than happy to have you find them!

[cracket]

I had no issues with downloading all 62k json files, now I have a huge folder with all of them and I'm trying to browse filtered subsets to understand how diverse are the results - like std. deviation, differences for the same sd reader and same card etc.. My question arisen because some other data-gathering portals prepare something like "aggregated download" - for example in your case it could be one zip file repacked weekly with all jsons inside. But this solution is rather if there will be more people interested in raw data. Anyway what's interesting is that still 32GB A1 cards are the fastest solution for running OS, all bigger (and newer) microSD cards are slower in terms of random read/writes, which surprises me. Which means I will not migrate my old Raspberry Pi to a bigger rootfs.

[TheRemote]

Hey cracket,

Those are actually some fascinating results! That's really interesting that 32GB seems to yield the best performance. That's exactly the type of thing I wanted to develop the benchmark to try to figure out. One of the big ones for me that it let me learn was just how slow regular USB flash drives are (even compared to application class SD cards).

The raw data is 3.1GB and counting and wouldn't make a lot of sense to export but I'm sure the meta_benchmarks table could be exported. That is just metadata generated from the raw tests and probably wouldn't be very large to release.

Once you've cached a benchmark it shouldn't really change after that point other than scraping new ones. There are some exceptions to this though such as if a model is fixed that I misidentified and things like that but the scores/raw numbers shouldn't change.

Fascinating results. I could absolutely figure out how to do this I'm sure. I could probably even write an API function that just dumps all of the benchmarks in one JSON that updates itself periodically. It actually worked this way before pibenchmarks.com existed and it was literally exported as a table and posted on my blog (very difficult to use once we got over a few hundred benchmarks).