ajaishankar
commented
9 months ago Also for json without quotes see the following link: https://deerchao.cn/projects/jsonlite/
Open ajaishankar opened 9 months ago
ajaishankar
commented
9 months ago Also for json without quotes see the following link: https://deerchao.cn/projects/jsonlite/
helloanoop
commented
9 months ago This is a perfectly valid use case in Bruno. This is not possible to represent in JSON as duplicate key value pairs are not allowed.
The lang should be indentation based as its important to clearly see where the contents begin, otherwise we will end up with having structure look like the latter.
body: {
type: 'xml',
// multiline is a backtick
data: `
<xml>
<name>Bru</name>
</xml>
`
}body: {
type: 'xml',
// multiline is a backtick
data: `
<xml>
<name>Bru</name>
</xml>
`
}We will support TOML as an alternative format to store requests along with Bru.
The way I look at it, TOML is kind of like CSS, and Bru is aimed to be SASS
PS: My plan was to move completely to TOML, but from feedback I have gathered, many folks like the Bru structure. The plan is to support both Bru and TOML
The goal of the lang standardization effort is not adoption by apps outside Bruno, but rather consistency and clarity of the lang itself. Think of it like this - openapi is a structure, and yaml is a lang What we have in .bru files today is a structure. The goal of the bru lang is to define the semantics for a consistent language, on top of which we can improve our existing structure.
There are also a lot of nuance, I will try to setup a 1hr call this week with the community to take feedback. I have made some good progress with the parser implementation https://github.com/brulang/bru-js
ajaishankar
commented
9 months ago The internal representation of multimap could be a key that has a array of values.
I feel TOML gives us a clean one to one mapping between the UI Table and the TOML Table
Indentation - we are mixing need for indentation with { [ ] } etc that make it unnecessary
Not really getting the distinction between structure and lang...
The call will help 👍
halostatue
commented
9 months ago I’ll speak against TOML as a human-readable configuration language, particularly once you get to arrays of maps. I originally introduced TOML for an internal project as our primary configuration language. Inasmuch as it was a "regularized" INI file format (simple dictionary), it is fine. The moment you start talking about anything more meaningful…it is a nightmare.
We quickly added both JSON and YAML support within the configuration parser because YAML was infinitely easier to read for complex structures with arrays of objects than TOML for the same values. JSON is used for minor things where the formatting does not matter, because ultimately standard JSON is not a human-friendly format. JSONC, HJSON are both superior for that, but they are still ultimately just JSON.
halostatue
commented
9 months ago As an additional note, if I were designing a language like this now, I would probably drop the colons on most of the object keys. They're not really necessary. Same with quotes on most strings. Like this:
http {
method GET
url https://www.usebruno.com/hello
headers {
Content-Type application/json
@disabled
Content-Type application/xml
}
array-value [ value1, value2, "value 3 with spaces" ]
"key with spaces" => "value with spaces"
body {
data '''
<xml>
<name>Bru</name>
</xml>
'''
}
}The only times that quotes would be needed are (1) the key or value is not in ^[-a-z_A-Z0-9]+$ and (2) it is a multi-line entry (and here I would fall back to ''' or """ like Python or Elixir rather than () as bru currently does). I would use => over : for special map expressions where "key" "value" is not sufficiently clear, but that is a weak preference mostly because of my background with Ruby and Elixir.
helloanoop
commented
9 months ago @halostatue I am liking your approach you outlined above (Inline with @ajaishankar aspirations - no quotes. He doesn't like annotations though :P)
As one of the advantages of Bruno is that the entire collection is able to be treated as a git repository, having contextually standard key order as well as a relatively compact textual representation is a huge benefit, and that's something that the "big 3" formats (json, yaml, toml) cannot viably provide, but a custom language as is already used can.
I like the way you elegantly summarized the need for .bru
I just did a quick comparison on how things would look. left - yours, right current Even without any syntax highlighting support, I feel the left side is mostly simpler, has less noise, and more information dense. @its-treason @mishushakov @MelGrubb would love to hear your thoughts too.
helloanoop
commented
9 months ago @halostatue
So, I am mostly convinced with your simpler approach that you proposed here: https://github.com/brulang/bru-lang/issues/2#issuecomment-1891276342
You have also convinced me to use ''' over ()
I think, one area that our viewpoints to differ is on indentation.
So here is why I think we should have strict indentation
.bru file you will see in the wildAlso curious to know if there is a strong rationale behind not using indentation.
Another way to come at this: Lets say we get to make yaml not need strict indentation, then would you vote on that ?
Based on your comment, I am assuming you wouldn't
YAML was infinitely easier to read for complex structures with arrays of objects than TOML for the same values.
mishushakov
commented
9 months ago TOML would not be my choice at all, because of the difficulties expressing nested objects. I like the Bru version without the quotes more. One point I'm not so sure about is leaving out the colons (:)
ajaishankar
commented
9 months ago It will help to define what all is it that we want to represent in a BRU file.
At present - a bru file specifies a request, some data/vars that is captured from the response, asserts and tests.
For these we do not need deep nesting at all.
res.body.$contains = Hello BruFeel the above would keep it simple and be able to be parsed line by line
It'll help if we take a really really complex bru file and explore more features like:
Just my two cents.
halostatue
commented
9 months ago I think, one area that our viewpoints to differ is on indentation.
Our viewpoints differ less than you may think. I am saying that calling bru-lang "indentation-based" is not entirely correct and complicates the parser. Having a preferred format with set indentation (making a bru fmt command with no configuration would be a fine idea).
bru-lang files will be more readable when formatted consistently. Reporting on indentation mismatches is good for a linter to present. Enforcing a specific style with a formatter is excellent. But the parser should not fail to parse a file because the indentation is non-standard (except, of course, for multi-line strings, which would be its own sub-parser).
YAML and Python are both languages where indentation is syntactically significant, so the parsing should fail if the indentation is not correct (or at least consistent; nothing about either language should require two or four space indentation, but blocks should always be indented at the same level and new blocks should be indented at a higher level).
I’m not a parser expert, and my only real expertise here is that I’ve shipped software written in ~40 different languages (not counting some DSLs that I have written that required no parsing because they were still Ruby under the hood), so I have a lot of exposure to different language specifications. I’m pointing out the indentation issue as a matter of the language spec. I would suggest that in addition to the basic language spec, which talks about how the parser itself should work, there would be a formatting spec which would describe what bru fmt (and related tools) would do.
If you like my 'quote'-less approach, I can put together a spec variant for that a bit later this week; I have quite a bit to do for the day job over the next couple of days.
helloanoop
commented
9 months ago I like the Bru version without the quotes more. One point I'm not so sure about is leaving out the colons (:)
@mishushakov Hmm.. I am kind of on the fence too on this. Colons (:) do play an important role in terms of more explictly/visually separating the key and value. JSON, YAML, TOML, etc. all use delimiters (: / =) to separate key and value.
For instance, lets see how package.json would look like in bru
As I stare at the two samples (with and without colons) I feel the one without colons is more elegant. But my brain is leaning towards having an explicit delimiter instead of relying on whitespace.
If you like my 'quote'-less approach, I can put together a spec variant for that a bit later this week; I have quite a bit to do for the day job over the next couple of days.
@halostatue That'd be great! Will see if I can setup a call this thursday on Discord. Ideally I would love to ratify this by end of this week.
MelGrubb
commented
9 months ago I posted a question in Discord earlier, but this is reminding me of some points I raised there: https://discord.com/channels/1036297729606963331/1039636949855645786/1196474782191988849
What technical limitations prevent using pure JSON where multi-line strings are represented by string arrays? It's not as pretty, and has a lot more quotes and commas in it, but it would mean that off-the-shelf JSON tools would have no problem editing the .bru files. I'm assuming they would retain the extension just to make it clear, although declaring a json schema at the top of the file would work just as well.
halostatue
commented
9 months ago I posted a question in Discord earlier, but this is reminding me of some points I raised there: https://discord.com/channels/1036297729606963331/1039636949855645786/1196474782191988849
What technical limitations prevent using pure JSON where multi-line strings are represented by string arrays? It's not as pretty, and has a lot more quotes and commas in it, but it would mean that off-the-shelf JSON tools would have no problem editing the .bru files. I'm assuming they would retain the extension just to make it clear, although declaring a json schema at the top of the file would work just as well.
From my perspective, there are multiple limitations:
To compare this in multiple formats, let’s take this variant of the format I proposed earlier (using colons instead of pure space delimited) and see what it would need to be in order to be safely stored as JSON:
http: {
method: GET
url: https://www.usebruno.com/hello
headers: {
// Try with application/json
@disabled
Content-Type: application/json
// The default format is application/xml
Content-Type: application/xml
}
array-value: [ value1, value2, "value 3 with spaces" ]
"key with spaces": "value with spaces"
body: {
data: '''
<xml>
<name>Bru</name>
</xml>
'''
}
}The JSON form would be longer by two lines, but would (a) be much harder to read, (b) have a much more complex required structure for anything which is allowed to have multiple values (e.g., headers, annotations), (c) lose any ability to have nicely formatted multi-line strings, (d) be restricted to UTF-8 only unless an encoded form were used that encoded non-UTF-8 data as UTF-8 safe strings (e.g., base64 or something like that).
{
"http": {
"method": "GET",
"url": "https://www.usebruno.com/hello",
"headers": [
{
"comment": "Try with application/json",
"annotations": [{ "name": "disabled" }],
"name": "Content-Type",
"value": "application/json"
},
{
"comment": "The default format is application/xml",
"name": "Content-Type",
"value": "application/xml"
}
],
"array-value": [ "value1", "value2", "value 3 with spaces" ],
"key with spaces": "value with spaces",
"body": {
"data": "<xml>\n <name>Bru</name>\n</xml>\n"
}
}
}When you add this with the fact that, as far as I can tell, a Go JSON serializer will randomize or alphabetize the keys whereas a Ruby JSON serializer will (mostly) order the keys on sort order, and I don’t have a clue what JavaScript would do, and you have something that would probably not have data stability for committing (and if manipulation is done with jq, you have to make sure that you never pass -c).
It may be that internally to Bruno, the parsed nodes of bru-lang would look a lot like what I have outlined above…but I don't think that it’s a good human-readable format.
Ultimately, it isn’t my call — but I do have an opinion on this and think that a custom format is both better and easier (especially since Bruno is already starting from having its own format+parser) than intentionally using one of the big three formats.
MelGrubb
commented
9 months ago Valid points. I hate the lack of comments myself. Sometimes things just need to be annotated. I get that JSON is a data format, and comments wouldn't round-trip well, but they really should have been allowed. They're not hurting anyone.
As for ordering, I'm an "alphabetize all the things" kind of guy myself. In the absence of a compelling reason to order things in a specific way, you should at lease fall back on something predictable. It makes git diffs so much better. I encourage/enforce letting ReSharper order the code on all my projects. It ensures that the only time two developers will have a merge conflict is when they both worked on things that would alphabetize to the same place in the file. It's reduced merge conflicts on many projects I've worked on.
halostatue
commented
9 months ago I have suggested that there be a bru fmt command and that when Bruno saves from the UI it does so in a standard way, but I do not believe that the order should be alphabetical as there is semantic meaning to be obtained from the construction of an HTTP request and validations.
MelGrubb
commented
9 months ago Alphabetical is the fall-back. For instance, in R#, I've "trained" it to order test classes differently. Methods that start with "Given" come first, followed by "When", and finally "Then". The "Then" methods still alphabetize, though.
For .bru, I'd say the meta information goes first, including docs, followed by the body, followed by the asserts/tests. Put the most interesting information first, and be kind of pseudo-chronological with the rest.
helloanoop
commented
9 months ago @halostatue I am building out the parser. Would love to know your take on this
The only times that quotes would be needed are (1) the key or value is not in ^[-a-z_A-Z0-9]+$
In a pair, there is a key and value
I think for key these should be the rules (as you suggested)
^[-a-z_A-Z0-9]+$I think for value these should be the rules
{}[],The above value rules apply to values in arrays too
For values, we shouldn't have the same restriction as key. Otherwise a lot of values in Bruno's context would have to be quoted
http: {
method: GET
url: https://www.usebruno.com/hello
header: {
Content-Type: application/json
}
body: {
type: xml
data: '''
<xml>
<name>Bru</name>
</xml>
'''
}
}Really love the version @halostatue presented above. Only thing I'm not so sure about is the ''' thing. Why not adopt JS' backtick (`) instead? The current version could be mistaken for a comment in Python.
halostatue
commented
9 months ago Minor update:
The only times that quotes would be needed are (1) the key or value is not in ^[-a-z_A-Z0-9]+$
In a pair, there is a
keyandvalueI think for
keythese should be the rules (as you suggested)
- need quotes if the value is not in
^[-a-z_A-Z0-9]+$I think for
valuethese should be the rules
- For unquoted strings, whitespaces around string are trimmed
- For unquoted strings, whitespaces inside the string are preserved
- need quotes if the value has these chars
{}[]:,
What would happen with these?
Content-Type: application/json // the server always wants application/json
Content-Type: application/json// the server always wants application/jsonI am assuming // for line comments, but it could just as easily be #. These are corner cases to be considered, but they can be solved in multiple ways:
json.// or # are special characters).Otherwise, with the one typo fix (value -> key), the rules look sensible and very similar to those of hjson.
@mishushakov the ''' approach is for multiple reasons:
""") from the amount of Elixir code that I write with multi-line strings and docstrings.body: {
type: xml
data: <<~XML
<xml>
<name>Bru</name>
</xml>
XML
}(Those are clearer because each block is its own possible name, but it has a massive negative impact on the parser complexity because it now has dynamic end markers.)
BTW, the ''' in Python is not a comment, but a docstring. It looks like a comment, but it will actually be stored by the compiled code as something for which documentation can be retrieved.
helloanoop
commented
9 months ago Do not allow in-line comments (easiest).
@halostatue I think this is best. The tradeoff is worth it imo
Also, for comments I feel # is cleaner instead of //
I'm not so sure about is the ''' thing. Why not adopt JS' backtick (`) instead?
@mishushakov
Given (`) and ''', my leaning is towards the latter
Reasons
helloanoop
commented
9 months ago How openapi spec would look in bru lang 🤩
Not saying that we should start representing openapi specs in .bru
But just an example to show how a markup lang with annotations improves readability so much. @ajaishankar I hope this convinces you on the usefulness of annotations
I also think we may be on to something here I would love to keep the lang as simple as possible. Anyone should be able to know everything about the lang by watching a 2 min video.
This is something that Json got right Yaml is too vast, like there are 9 ways to represent multiline strings. That is spec bloat imo TOML shouldn't have standardized datetime stuff in the lang
MelGrubb
commented
9 months ago Regarding the triple-single-quotes for multi-line strings. I equate this to the "raw string" feature of modern C#, so I like it, but would suggest one thing. I am used to how raw strings use the position of the quotes to clarify the indentation of the following lines. Your example reads like this:
... body: { type: xml data: ''' <xml> <name>Bru</name> </xml> ''' } }
But the way C# has implemented raw strings, it would look like this, with the ''' on its own line:
... body: { type: xml data: ''' <xml> <name>Bru</name> </xml> ''' } }
It's just a thing I got used to seeing, and now makes sense. I know what the indentation of the first <xml> should be because the position of the opening ''' is meaningful. It visually establishes where the left margin is. In fact, that entire block could be indented one step further in like this, making its relationship as the child of the "data" element even more clear.
... body: { type: xml data: ''' <xml> <name>Bru</name> </xml> ''' } }
In the end, this is a data format, and I'm not going to interact with it directly most of the time. I'll be using the editor inside Bruno, so this is largely invisible to most users, and if one way is easier to implement, then do that, but I thought I'd throw this in there. I saw an example of these kinds of strings in C#, and their choices make sense to me.
halostatue
commented
9 months ago This should be accommodated in the spec changes that I am putting together in #3 where it says that the indentation is relative to the line where the opening ''' appears. If the parser ignores whitespace and newlines after a key delimiter (:) until it gets to a value, then it should not matter whether the ''' appears on the same line or the next line (and if on the next line, it should be indented one level from a formatting perspective).
The question ends up being how a Bru parser should handle this:
{
key:
second-key: []
}If the result is {key: null, second-key: []}, then I don't see a way to put ''' on the following line. If the result is a syntax error, then ''' can be on the following line. I favour explicit over implicit, so I think it should be a syntax error — but that may complicate the parser, and I will defer to @helloanoop as the primary parser implementer.
helloanoop
commented
9 months ago I have a parser implementation running here https://bru-playground.vercel.app/
The implementation is currently indentation based.
There are still edge cases that needs to be polished and fixed.
ajaishankar
commented
9 months ago Since we are trying to build a full fledged language plus annotations, would like to share the following
I still am of the opinion that TOML (with dotted keys) will be more than enough to represent a bru file.
That is instead of
[http]
method = "get"
[http.headers]
authorization = "Bearer {{token}}"just have
[http]
method = "get"
headers.authorization = "Bearer {{token}}"Just my (another) 2 cents 😄
helloanoop
commented
9 months ago I personally like TOML and even spent a whole week to think how we could represent everything (including all edge cases) I even started writing the toml parser - https://github.com/usebruno/bruno/tree/main/packages/bruno-toml/tests/headers
When I presented it to the community (we had 15 people on call), the feedback was - "hey not all of us like toml - you should actually make bru work". Feedback on forums were also favourable towards bru structure (which you and I designed in 2021 and has surprisingly scaled without significant hiccups)
And after mostly working on standardizing bru this week with an almost completed parser, I believe standard bru is actually better and must be the canonical choice to work with Bruno
We however will support using toml and yaml parsers in the future and allow people to pick the lang as per their preference (just like openapi allows you to choose json/yaml)
UCL - Does not support multimaps Swagger Annotations - These are supported in languages but not in (json/yaml) itself.
OK, so how you would represent headers that are
@ajaishankar how would you represent the below perfectly valid api request in TOML
In Bru, it would be (16 lines including newlines)
http: {
headers: {
Content-Type: application/json
@description('XML is also supported')
@disabled
Content-Type: application/xml
disabled: 'good luck 😉'
description: 'bru is awesome 😄 '
@enum(active, inactive, gotcha)
enum: active
}
}Here you go, one of the many ways to skin the cat 😉
[http]
headers.content-type.0 = 'application/json'
headers.content-type.1 = 'application/xml'
headers.disabled = 'good luck 😉'
headers.description = 'bru is awesome 😄'
headers.enum = "active"
[annotations]
headers.content-type.1.description = "XML is also supported"
headers.content-type.1.disabled = true
headers.enum.values = ["active", "inactive", "gotcha"]A "multimap" in UCL automatic arrays is nothing but repeated keys.
content-type = 'application/json'
content-type = 'application/xml'A "multimap" in UCL automatic arrays is nothing but repeated keys.
Hmm.. Then the only thing would be lack of annotations I guess.
[http]
headers.content-type.0 = 'application/json'
headers.content-type.1 = 'application/xml'
headers.disabled = 'good luck 😉'
headers.description = 'bru is awesome 😄'
headers.enum = "active"
[disabled]
headers.disabled = true
[annotations]
headers.content-type.1.description = "XML is also supported"
[enums]
headers.enum.values = ["active", "inactive", "gotcha"]I believe disabled and enum should not go into annotations. So I have moved it to a seperate block.
Now, what if you have a header that is called enum.values
Also .0, .1 for storing multimaps is "magic" (not my word, its what the community called the disabled flag in bru ~ )
Now just putting the same bru block for a comparision.
http: {
headers: {
Content-Type: application/json
@description('XML is also supported')
@disabled
Content-Type: application/xml
disabled: 'good luck 😉'
description: 'bru is awesome 😄 '
@enum(active, inactive, gotcha)
enum: active
}
}IMO the bru structure is
mishushakov
commented
9 months ago @ajaishankar the problem is that this format is not very legible and would confuse people when diffing in git
ajaishankar
commented
9 months ago @mishushakov not sure about the lack of readability of the format and the diff since TOML is supposed to be human readable.
But I do get what @helloanoop is saying about colocation of the annotation.
So I guess what we are ending up with is JSON with
If so let me throw another option to consider - the @key instead of annotation! 😉
http: {
headers: {
Content-Type: application/json
@description: XML is also supported
@disabled: true
Content-Type: application/xml
disabled: 'good luck 😉'
description: 'bru is awesome 😄 '
@enum: [active, inactive, gotcha]
enum: active
}
}Which will allow for complex annotations like
@retry: {
maxAttempts: 3,
backoff: {
delay: 1000,
multiplier: 2
}
}
http: {
method: GET
}@mishushakov not sure about the lack of readability of the format and the diff since TOML is supposed to be human readable.
The key phrase for TOML is "supposed to be". For simple configuration / data structures, TOML is simple. The moment you start adding either repeated structures (arrays) or object depth greater than two levels and TOML is far less readable.
As I said earlier, I had adopted TOML as our configuration format for a work project and it worked very well right up until the point that it didn't. We still use it, but have added post-TOML parser processing for things like key = $import(filename.type) and $import = filename.type where we read YAML or JSON for deep tree data structures that are quite painful to express in TOML.
I’m still doing some work with the new spec (#3), but one of the things that I’m going to try to do is express what the simplest representation of the current semantics would be in JSON, YAML, and TOML (so that application/bruno+toml is a meaningful media type, once registered). That will probably be a separate file from the primary spec and the README — and is still ultimately up to @helloanoop and the rest of the Bruno team as to whether the proposed "standard" representations (which is different than parser representations would be) are what is desired for interoperability.
I’m going to likely cheat on the JSON and TOML representations and build out an equivalent in YAML and then just allow remarshal to do the work for me (yaml2toml, yaml2json) for the less-friendly formats.
ajaishankar
commented
9 months ago @halostatue I don't disagree.
My only question is what exactly will be in a .bru file?
If the goal is to represent a request body as pure json/toml/bru etc, yes it will be complex.
But if all we are doing is pasting some JSON in the Bruno editor and that gets written in the .bru file as a multiline string, then the concerns about nesting and arrays no longer hold true.
halostatue
commented
9 months ago @halostatue I don't disagree.
My only question is what exactly will be in a .bru file?
If the goal is to represent a request body as pure json/toml/bru etc, yes it will be complex.
But if all we are doing is pasting some JSON in the Bruno editor and that gets written in the .bru file as a multiline string, then the concerns about nesting and arrays no longer hold true.
I think that adequately representing what can be in a request — including a request body — is absolutely complex, especially as Bruno allows for configurable requests and exploratory testing.
In Insomnia, I have a request where I have two sets of headers already entered. When I need to make that request for client A, I check one set of headers. When I need to make that request for client B, I check another set of headers. Could I make this two separate named requests? Yes, but then if I need to update something in the payload, I now have two requests to update.
So, nesting and arrays absolutely hold true because of the locality rules that @helloanoop pointed out (and those locality rules are doubly important from a git review perspective, as it reduces the churn).
To me Bruno should have a human-comprehensible text format that is reasonably stable for use with shared request VCS repositories. By comprehensible, I do not just mean that a human can read it (TOML can do that), but that I should be able to fairly easily see what the saved request does just the same as if I were viewing it in the UI. Practicality suggests that YAML is going to be the best choice of existing formats, JSON the easiest choice of existing formats, and a custom serialized format the densest choice. TOML does not even enter the question beyond a certain small amount of complexity, which for me starts with the use of any array structure (multimaps are essentially arrays of objects).
ajaishankar
commented
9 months ago Request body IMO may not need a structured representation, since it could be
For the different set of headers per client, it's how Bruno chooses to represent that.
To me the following overridable list of headers per client seems ok.
[headers]
[headers.client-1]
[headers.client-2]I feel there is a 1-1 mapping between a table in Bruno UI and a table in TOML, but gonna leave it to the community...
helloanoop
commented
9 months ago Bru Lang Website is Live: https://www.brulang.org 🎉 🥳
The current version of bruno omits quotes and nesting.
If we introduce both, it is really no different from either json (without quotes for keys) and exactly a JS object (sans commas)
The current proposal has the following features:
Which can again be represented as a simple JS object.
If the objective is to be a human readable config and also have a 1-1 mapping between the bruno UI, the following might serve us better
A simpler version of TOML
Or put status.meta to a different section