0.2.0 Milestone

[max-sixty]

I'm not sure if this is literally "0.2.0", or we should have intermediate releases, but this is the next big milestone!

My sense is that to get to the next level, PRQL needs to "get out there" — getting feedback from users, demonstrating that it can solve real pain points, exploring ways to hook into real workflows.

For that to happen I think we need:

• A website with live updating PRQL<>SQL — https://github.com/max-sixty/prql/issues/271.
  • I worked on this last weekend (though wasn't that productive, took a long time to get to the current version)
• Much better language documentation https://github.com/max-sixty/prql/issues/232. I have been intending to address this for a few days now...
• Window functions https://github.com/max-sixty/prql/issues/300 — more than half the people I speak to about PRQL tells me that this is painful in SQL and would be a great feature to add.
• More language features — there's lots of small annoyances in SQL I think we could solve without Herculean effort. Malloy have done this very well https://github.com/max-sixty/prql/issues/71. Some things that come to mind; feel free to open specific issues to discuss them:
  • f-strings — CONCAT(foo, " ", bar) -> f"{foo} {bar}"?
  • Regex — REGEX_MATCH(foo, "\\w{3}") -> foo ~ r"\w{3}"? Or -> regex foo r"\w{3}"?
  • Ranges — BETWEEN 1 AND 3 -> in 1..3?
  • Dates — "2021-01-01" -> @2021-01-01? And DATE_TRUNC(foo_date, YEAR) -> foo_date.year? Or -> foo_date | as year? Or foo_date | to year?
  • Offsets — DATE_ADD(DATE "2008-12-25", INTERVAL 5 DAY) -> @2008-12-25 - 5day?
  • Case / when / then
  • Any others?
  • TBC, I don't think we have to do all of these — and getting more feedback from users will help us target the most important ones. And if we want to change something later, that's fine too — we're early, and we'll have versioning soon anyway.
• Building out the standard library a bit https://github.com/max-sixty/prql/issues/217 — if we had a systematic way of doing this by parsing some docs (e.g postgres) and selecting what we want to include / exclude, that could be very efficient. Otherwise gradually adding things could be a great way for a new contributor to make an impact.
• [Edit, based on @aljazerzen comment below]: Logo & org https://github.com/max-sixty/prql/issues/303

Other big things that I don't think we need to do, but were in my mind when thinking about this:

• Arrays / Structs. I think this is quite awkward in SQL and we could make it much easier. But one issue is that the DB implementations are different IME.
• Dialect & versioning — prql version:0.1 db:snowflake. As @aljazerzen points out in https://github.com/max-sixty/prql/issues/292#issuecomment-1086685010, we'll have to start having different implementations for different dialects. But I'm not sure that's required to get people experimenting with the language
• JSON
• Database cohesion — https://github.com/max-sixty/prql#database-cohesion — this would be awesome and @qorrect has demonstrated that it's feasible. But it's more difficult, requires context, and more difficult to test.

We had a phenomenally successful push when we first hit Hacker News etc, even with only a proposal — we got a huge amount of high-quality feedback and a wonderful group of contributors. So I'm hoping we could have another push with people actually using the language, if we could get to this stage.

Is there anything missing there / anything we could remove?

[aljazerzen]

Except for a few smaller language features, I agree on the list of required things. I would also add dialect and versioning to the list, because currently most of queries won't work on Postgres due to TOP/LIMIT.

The best approach to find out what's needed is to actually go a try to use PRQL ourselves - as I tried to translate a few queries, I found out the problems with AVG(AVG(x)) which lead me into the rabbit hole. I think that before 0.2 and when we put software into people's hands, we have to test it on real-world use-cases. Try to find some SQL queries we've written or a course in advanced SQL and try to translate it into PRQL.

And web thing: absolutely. Much more people will check out the project if they can test it on-the-fly, without downloading binaries or compiling Rust.

And to add one thing: logo and general design of the web site.

[max-sixty]

The best approach to find out what's needed is to actually go a try to use PRQL ourselves - as I tried to translate a few queries, I found out the problems with AVG(AVG(x)) which lead me into the rabbit hole.

I very much agree! And this is an underrated way for people who want to contribute but don't know rust — these sorts of error reports are really useful.

into the rabbit hole.

Or towards the light :)

And to add one thing: logo and general design of the web site.

Thanks, added!

[max-sixty]

Comparing our progress to the list above, we're in a really good place.

Some small things outstanding:

• review of the docs — it's much better but there are still a couple of cobwebs is there (on me, though any help welcome)
• stdlib
• promote prql-js to reference docs (in place of prql-web, rip)
• anything else?

Apart from that, some things we should decide whether we wait for:

• Slicker website
• Inline errors in online compiler (i.e. more prominent message, squiggly line under text)
• syntax highlighting in online compiler
• Integrations, some that are in progress or have been discussed:
  • Jupyter
  • dbt
  • Prefect
  • Rill?
• Blurbs from people who want to support PRQL

I had thought we should wait, but I'm less sure now. For the integrations, if I can jam one of these out in a weekend (maybe the Prefect one would be easiest?) then it's worthwhile, but otherwise we may be Waiting for Godot, and getting out there would get the project more visibility, feedback, and contributors.

OTOH I worry that we're in a place where people will "star the repo and forget about it", if they can't start using it. If there were a better way of people saying "I like the idea, keep me updated", then that would mitigate this, something stronger than #1 but less heavy than joining the discord.

Very open to thoughts. What would people prioritize pre 0.2?

[aljazerzen]

First of all, I think that it would be better to focus with our milestones:

• 0.2 milestone can be the "it works, it won't change significantly, you can check out the language on the web" release and
• 0.3 will be "it's polished, you can checkout the tooling on your databases"

I do think people will "star and forget", but more people will "star and remember" if we do more smaller milestones. Also, a lot of people have missed PRQL on it's first milestones. We need to get feedback on changes we've done and how hard it really is to learn the language.

With this in mind:

• docs are far ahead of what we had a few weeks back, great work @max-sixty ! But to make it dummy-proof, I would ask someone who has not read the reference, to read it and try to learn the language.
• stdlib may not be the most important. For most queries, it's quite enough and I don't think that people will drop the language because of it, if we mention that stdlib is still in the works. We add its documentation, though.
• website needs a designer. A mockup at least. But I'm not confident that we need to in 0.2. What we do need, is a better editor experience. Maybe, we should make a separate web application with React/Angular that:
  • does not need to load when opening the landing page of the project,
  • saves your queries,
  • has syntax highlighting,
  • has inline errors,
  • has many pre-saved examples that you can load.
• integrations are not key here, one can learn the language on the web editor.

So focus for 0.3 would be:

• integrations, all of the above
• website
• type checking, function currying, small quality-of-life features
• maybe: prql-fmt / prql-language-server

[max-sixty]

Hoping to release tomorrow, Tuesday. No firm guarantees as I keep finding small things; but we will release this week in whatever state we're in.

Remaining things:

• Release pyprql; required for dbt-prql
• Add dbt-prql docs to our docs
• Add prefect docs (very simple)
• Roadmap — update / remove?

[max-sixty]

This is releasing tomorrow!