Add support for URL parsing to standard library functions (or add String::indexOf)

clintgibler commented 4 years ago

Hey! First off, thanks for building Darklang! It's a really exciting project, and I'm looking forward to seeing where things go 👍

Problem

Given a URL as a String, I'd like to parse the URL into its component parts.

For example, I'd like to be able to turn https://example.com/foo/bar?arg1=some_val&arg2=other_val#somehash into:

domain: example.com
path: /foo/bar
url_parameters: {"arg1": "some_val", "arg2": "other_val"}
hash: #somehash

Many languages (e.g. JavaScript) have standard library functions that make this easy.

What I've Tried

I reviewed the full Darklang API docs, but did not find any functions under a URL:: namespace.

Then, I decided to try to hack something together using language primitives, e.g. direct string manipulation. Unfortunately there is no RegularExpression or Regex namespace, so I reviewed String and List.

Interestingly, because String::contains(lookingIn, searchingFor) returns a Boolean and not an Int corresponding to the first location of searchingFor, and there is no String::indexOf(stringToSearch, characterOrString), it does not appear possible to manually extract what I want from a String URL. (Unfortunately String::first and String::last do not appear to help here either)

I then thought perhaps I can convert the String to a List with String::split url "", however, List::findFirst returns the first match, not the index of the match.

HTTP handlers can already parse URLs, so I thought maybe I could create an HTTP handler that I pass in the target URL, and in the route I could destructure the URL and return that to the caller. However, the URL would be a parameter not the URL matched by the route, so I don't think that can work.

Potential Solutions

Add `URL::` to Standard Library

This may be more work, but I think it's overall the best UX: writing custom code to parse URLs is annoying, error prone, and full of edge cases.

HTTP handlers already do URL parsing, so hopefully there's already internal code that does this, and you could just expose the URL parsing code as standard library functions.

As Darklang aims to give users a first class web development experience, it seems likely that other users will build server-side functionality that requires parsing URLs, so this seems (to me) reasonable to include in the standard library at some point.

Add `String.indexOf` (and maybe `List.indexOf`)

Rather than adding a handful of URL helper functions, you could add these and let users code URL parsing logic themselves.

Further, at least in my opinion, these API functions seem like core language functionality that users will need at some point regardless, so probably worth adding when time permits.

clintgibler commented 4 years ago

Sorry for the wall of text 😅

Any thoughts you have on how you'd approach accomplishing what I'm aiming to do, and/or the relative difficulty of various solutions, would be much appreciated 🙏

pbiggar commented 4 years ago

Thanks for the report! Yes, this seems great, and a few folks have suggested it before. Can you provide a suggested signature for the function?

clintgibler commented 4 years ago

Ah great! To me, I'd imagine the signatures looking something like this.

`String::indexOf` and `List::indexOf`

String::indexOf (Str string, Str searchFor) -> Int Returns the first index of searchFor that occurs with string, or -1 if searchFor is not found within string.

Example usages:

String::indexOf("foobar", "foo") => 0 - assumes Strings are 0-indexed
String::indexOf("foobar", "o") => 1 - Returns the first match when multiple are present
String::indexOf("foobar", "bar") => 3
String::indexOf("foobar", "baz") => -1 - When searchFor is not present

I could imagine some use cases where one might want all of the matches in string, not just the first. Not sure if that's worth adding to the language as well.

List::indexOf (List list, Any searchFor) -> Int - Basically the same behavior as String.indexOf.

`URL::`Standard Library

let url = "https://example.com/foo/bar?arg1=some_val&arg2=other_val#somehash"

URL::domain(url) => "example.com"
URL::path(url) => "/foo/bar"
URL::searchParams(url) => {"arg1": "some_val", "arg2": "other_val"}
URL::hash(url) = "#somehash"

pbiggar commented 4 years ago

These are great, thanks!

Do you want to add them? It's quite straightforward, with a good example here: https://github.com/darklang/dark/pull/2749/files

pbiggar commented 4 years ago

Contributor docs are here: https://docs.darklang.com/contributing/getting-started/

The URL parsing library you can wrap (it's already available as Uri, I think): https://docs.mirage.io/uri/Uri/index.html

clintgibler commented 4 years ago

Thanks for the links, that's quite helpful!

Seems feasible, though not sure when I'll have time to work on it unfortunately. We'll see :)

pbiggar commented 3 years ago

FYI: the new backend (in F#) will remove some of the obstacles to having a regex library too.

clintgibler commented 3 years ago

Oh nice, that'd be great!

pbiggar commented 1 year ago

URL should be in the package manager

darklang / dark