Guide: Write a guide about Batching

tusharmath commented 8 months ago

Write a guide about the batching capabilities of Tailcall. Consider a food delivery application where a lot of users are requesting the same data. Batching could be implemented based on GEO Location / City / Locality etc. Emphasize on the impact it has on performance and how it can help scale. Consider the backend to be in REST.

Content Creation Requirements

To maintain the quality of our content, please adhere to the following guidelines:

1. Accuracy in Language

Grammar and Spelling: Ensure your content is free from grammatical and spelling errors.

2. Tone and Style

Neutral Tone: Maintain a neutral, non-emotional tone throughout the content.
Engaging Style: Write in a free-flowing and engaging manner, keeping the reader's interest.

3. Content Integrity

Fact-Checking: Verify all information. If unsure, use Discord to clarify.
Relevance: Ensure all content is cohesive, to the point, and directly related to the title.

4. Originality

Avoid Low-Effort Content: Content should be original and not solely generated by AI tools like ChatGPT.

PS: Adherence to these guidelines is crucial. Content not meeting these standards may require revision or may not be accepted.

tusharmath commented 8 months ago

/bounty 120$

algora-pbc[bot] commented 8 months ago

💎 $120 bounty • Tailcall Inc.

Steps to solve:

Start working: Comment /attempt #882 with your implementation plan
Submit work: Create a pull request including /claim #882 in the PR body to claim the bounty
Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

🙏 Thank you for contributing to tailcallhq/tailcall! 🧐 Checkout our guidelines before you get started. 💵 More about our bounty program.

ibishal commented 8 months ago

can i get assigned?

ayewo commented 8 months ago

Interested in working on this ✋.

ayusham001 commented 8 months ago

@tusharmath please assign it to me

nxshanth13 commented 8 months ago

@tusharmath brother, I'm interested to work on this.

FaizanShaikh-3033 commented 8 months ago

Assign it to me, i can do it.

tusharmath commented 8 months ago

Have not seen any progress on this issue, unassigning it from @ayewo, let me know if someone wants to work it.

ayusham001 commented 8 months ago

@tusharmath i would love to work on that Please assign it to me

ayewo commented 8 months ago

@tusharmath was about open a PR ...

While writing this, I ran into some gaps in Tailcall's implementation of batching, and more generally, about how the @http operator works.

We need a jsonpath attribute to be added to the @http(...) operator so it is easy to index inside JSON responses returned by upstream endpoints.
- 1a. The https://jsonplaceholder.typicode.com/users URL uses a top-level array for its JSON and it's response is shaped like this:
```
[
{...},
...,
{...}
]
```
- 1b. This URL https://dummyjson.com/users doesn't use a top-level array for its JSON due to its inclusion of pagination attributes so it's JSON is shaped like this:
```
{
"users": [{...}, ..., {...}],
"total": 100,
"skip": 0,
"limit": 30
}
```

This difference in output is why the 2nd URL fails in Tailcall with the following error i.e.

{
  "data": null,
  "errors": [
    {
      "message": "internal: expects an array",
      "locations": [
        {
          "line": 33,
          "column": 3
        }
      ],
      "path": [
        "users"
      ]
    }
  ]
}

Once implemented, we could use a splat expression[^splat] inside the jsonpath to target the data we need:

  type Query {
  users: [User]! @http(path: "/users", jsonpath: "users.[*]")
}

Batching only supports use of keys that appear in the query params.
- 2a. Batching works with query params URLs: https://example.com/path/users?id=1&id=4&id=7 where the query param id is also used for groupBy i.e. @http(path: "/users", query: [{key: "id", value: "{{value.userId}}"}], groupBy: ["id"])
- 2b. Batching doesn't work with multiple path params URLs: https://example.com/path/users/1,4,7 where the path params implies bulk fetch 3 users with the IDs 1, 4, & 7 i.e. https://example.com/path/users/1, https://example.com/path/users/4, and https://example.com/path/users/7).

ipstack.com is an example of an API designed to only support multiple path params for bulk requests.[^note2]

Batching on ipstack.com is done by comma-separating multiple path params when constructing a bulk request:

http://api.ipstack.com/134.201.250.155,72.229.28.185,110.174.165.78
    ? access_key = YOUR_ACCESS_KEY

When called from a schema @http(baseURL: "http://api.ipstack.com", path: "/{{value.ip}}", groupBy: ["ip"]), Tailcall assumes the request is missing query params so it fails with the following error:

{
  "data": null,
  "errors": [
    {
      "message": "IOException: Unable to find key ip in query params",
      "locations": [
        {
          "line": 39,
          "column": 5
        }
      ]
    }
  ]
}

[^splat]: Splat expressions in HCL

[^note2]: Similarly ip-api.com, ipdata.co and ipinfo.io are 3 geolocation services whose batching endpoint will not work with Tailcall as it can only be done over POST. Tailcall only supports batching over GET requests. When I tested ip-api.com in a schema via @http(baseURL: "http://ip-api.com", path: "/batch", method: POST, body: ...), the Tailcall server failed with Error: Invalid Configuration caused by GroupBy is only supported for GET requests.

AbdoALPOP commented 8 months ago

👋 Hey Tushar,

I am interested in your task and available to start immediately.

I am experienced with Docusaurus.io, Nextra, Mkdocs, and markdown. I can provide you with a user-friendly guide.

Here are some of my live guides:

I'm looking forward to hearing from you soon 😃 Contact me and let's get started.

github-actions[bot] commented 7 months ago

Action required: Issue inactive for 30 days. Status update or closure in 7 days.

ayewo commented 7 months ago

.

github-actions[bot] commented 6 months ago

Action required: Issue inactive for 30 days. Status update or closure in 7 days.

ayewo commented 6 months ago

.

github-actions[bot] commented 4 months ago

Action required: Issue inactive for 30 days. Status update or closure in 7 days.

ayewo commented 4 months ago

.

ayewo commented 3 months ago

.

tusharmath commented 3 months ago

/tip 20$

tusharmath commented 3 months ago

@ayewo Apologies, this has taken too long. Unfortunately, I don't think this in line with what we expected, and the to and fro on the discussion did not seem like is going anywhere. The content is too long and sort of distracts us from the main topic. I appreciate you putting time in creating this PR and hope to see more from you in the future 🙌

Once again, sorry for not responding sooner.

ayewo commented 3 months ago

@ayewo Apologies, this has taken too long. Unfortunately, I don't think this in line with what we expected, and the to and fro on the discussion did not seem like is going anywhere. The content is too long and sort of distracts us from the main topic. I appreciate you putting time in creating this PR and hope to see more from you in the future 🙌

Once again, sorry for not responding sooner.

@tusharmath well, I tried to suggest on Discord, two months ago, that Tailcall pay for the content produced so far since everything was taking to long already …

Glad you finally decided to do this but the tip is really 😏. Can you please make it $50?

Also, you’ll need to include the GitHub username for the tip command to work from here (or do it directly on the PR without the username as you did above) i.e.:

/tip $50 @ayewo

tailcallhq / tailcall