URL parsing API - Githubissues

robingustafsson commented 5 years ago

Problem

Parsing URLs is a recurring need when working with HTTP and other protocols. There's currently no native API in k6 for so. As https://github.com/grafana/k6/issues/991#issuecomment-767493866 mentions a jslib polyfill is available but a native API for a context like k6, it would be better.

Proposal

As the comment history suggests, we would like to implement all or most of the relevant parts for k6 of the URL interface as defined by the Web API.

import URL from "k6/url";

const myURL = new URL("https://user:pass@sub.example.com:8080/p/a/t/h?query=string#hash");

console.log(myURL.hostname);

import { URL, URLSearchParams } from "https://jslib.k6.io/url/1.0.0/index.js"

const address = new URL('https://test.k6.io')
address.searchParams.append('hello', 'world')
console.log(address.toString()) // prints https://test.k6.io/?hello=world

const search = new URLSearchParams([
  ['foo', 'bar'],
  ['hello', 'world']
])
console.log(search.toString()) // prints foo=bar&hello=world

robingustafsson commented 3 years ago

I agree that the introduction of https://github.com/loadimpact/jslib.k6.io/tree/master/lib/url/1.0.0 solves most of this issue. The only reason why I'd still argue this should be included in k6 core is that URL parsing/composing feels like a very basic and core functionality of a load testing tool 🙂

Another argument is that we have users who need to run k6 in firewalled environments where no Internet-bound connections are allowed and thus using jslib.k6.io hosted libs becomes more of a involved process of having to download this kind of lib and use it as a local dependency. Not a major issue perhaps, but for sure inconveniant for such a basic (in the context of a load testing tool) thing as composing and parsing URLs.

na-- commented 3 years ago

Fair enough, very good points! And taking a second look at the JS polyfill, 32kb of minified JavaScript for such a simple thing seems a bit excessive... :scream:

So, unless that can be substantially reduced, we probably don't want to bundle it and have every VU parse these 32kb when it's initialized, now that we're so close to finally getting rid of core.js completely... So my vote is to write it in Go and try to upstream it in goja, but if that doesn't get accepted, we can keep it in k6, since you are right that as a load testing tool, it's not going to be out of place. In any case, the Go implementation should be relatively easy, we can base it on the net/url package - hopefully here aren't any big differences.

na-- commented 3 years ago

FWIW, people are complaining about the weight of the URL polyfill from core.js (https://github.com/loadimpact/k6/issues/1851), so another data point in favor of having a native Go version of it.

mstoykov commented 3 years ago

again ... I doubt the polyfill is heavy ... at least it wasn't for me ;)

TimDaub commented 2 years ago

I was disappointed when I learned that k6 is implemented in go.

E.g. that I get an error on URLSearchParams is confusing. In node, I'd expect compliance with e.g. whatwg specs. For k6 I don't know what to expect. How can I solve this problem?

[0000] ReferenceError: URLSearchParams is not defined                                             upted itat getParam (
file:///Users/timdaub/Projects/strikedao-stress-test/script.mjs:37:20(3))         iters  at register (file:///Users/timdaub
/Projects/strikedao-stress-test/script.mjs:59:22(40))

na-- commented 2 years ago

@TimDaub, for now, by importing the polyfill linked above or some other URL JS polyfill.

As you can see from the comments in this issue, we want to have this specific API natively in k6 eventually, since it's so useful for our use case. However, I doubt we'll ever have full native compatibility with all WHATWG specs, since k6 is not a browser... node.js doesn't have that either! For example, it doesn't support the Fetch API, right? It has its own way of making HTTP requests that is different from browsers and from k6.

In the long run, we want to have full ECMAScript compatibility in k6 (though we have a lot of work remaining there, see https://github.com/grafana/k6/issues/2296#issuecomment-993356717), and we want to support the Web (WHATWG) APIs that make sense for our use case.

Gikkman commented 3 months ago

The recommended approach here yields a warning with the latest version of k6:

Save the following as script.js

import http from 'k6/http';
import { URL } from "https://jslib.k6.io/url/1.0.0/index.js"
import { sleep } from 'k6';

export const options = {
    vus: 1,
    duration: '10s',
};

export default function () {
    const appUrl = new URL('http://example.com');
    http.get(appUrl.toString())
    sleep(5);
}

Then run it through docker:

$ docker run --rm --net=host -i grafana/k6 run - <script.js  

time="2024-05-28T11:57:33Z" level=warning msg="The \"wrong\" path (\"file:///\") and the path actually used by k6 (\"file:///home/k6/\") to resolve \"https://jslib.k6.io/url/1.0.0/index.js\" are different. Please report to issue https://github.com/grafana/k6/issues/3534. This will not currently break your script but *WILL* in the future, please report this!!!"

grafana / k6

URL parsing API #991

Problem

Proposal

Further reading