kdembler
commented
2 years ago I think you did a good job capturing the most pertaining issues. Biggest pains from my perspective:
1. Unreliability
One issue is the QN going down entirely, another issue are bugs, more or less visible. Sometimes things will be working fine but you add one specific field to the query and the request fails. Those are often found mid-development, blocking the developer.
2. Restricted filtering
The fact that we cannot do filter like where: { video: { channel: { id_eq: "xxx" } } } because it's one level too deep. This restricts us quite often and for Rhodes resulted in probably 10+ requests for QN developers to "flatten" fields on different events and entities so that they can be consumed. Also, lack of basic NOT filtering is problematic sometimes
3. Hydra weirdnesses
We had to work around a couple of Hydra issues when working on Rhodes, mostly related to variants and relations. Current schema for NFT should be a good example:
type OwnedNft {
# ...
transactionalStatus: TransactionalStatusIdle | TransactionalStatusInitiatedOfferToMember TransactionalStatusBuyNow
transactionalStatusAuction: Auction
}We keep accumulating those workarounds.
4. Incident Management
I think what you brought up is a very valid point, that we don't have much visibility into when things break. For Atlas, this wasn't that big of an issue since incidents for us happened mostly in development, so usually a developer noticed QN not syncing right away. However, it's been an issue to get access to the QN to get logs (this has been mostly alleviated at this point). There is also the fact of sync taking quite a bit of time to finish. It's not an issue assuming we don't do it often, but during Rhodes development there were days when I did 3 re-syncs. If each takes 2 hours to complete, it adds up.
zeeshanakram3
ondratra
mnaamani
Question
Should we do this meeting sooner? Depends on how soon people can get their deliverables ready, see below.
Meeting
Tue 10thParticipants
Background
The Query Node (QN) is an instrumental piece of the infrastructure that not only powers key applications in the Joystream ecosystem, namely Pioneer and Atlas, but also powers key secondary infrastructure software like storage and content delivery nodes, as well as the CLI and other t
Current Problems
Here is an incomplete and unranked list of problem that seem to occur as part of the development and usage of the QN.
Note: This list reflects my point of view, which is not very granular or complete, and so it may need adjustments before the meeting.
1. Unreliable
It very frequently goes down at various stages of both development and in production. In particular bugs during development are very costly, as teams relying on it have to wait around - perhaps mock APIs, while a QN developer can take a look, fix the problem and redeploy.
2. Impotent API
It cannot, as a matter of Hydra constraints, offer the APIs that either client teams prefer, or absolutely need as a matter of basic product features. This often requires complex coordination about what product features are required vs. nice to have, and exactly how to work around limitations which are unavoidable. Frequently we now have to build redundant state into the API, which means mappings become harder to write correctly, further contributing to the prior problem.
3. Synch issues
This one may just be me misunderstanding
On occasion it seems that we have had problems caused by making mistakes in deployments, in particular from where to process for a given purpose.
4. Incident Management
It seems we rarely have errors detected by our own Jsgenesis personell, it's always the community which finds the problem and then has to alert someone on Discord, and then whomever catches this problem ends up having to try to get to whomever is able to look into the problem.
5. Debugging
It seems it does take quite a bit of time determinig why exactly the QN went down ex-post.
Meeting Purpose
Map out and prioritise what changes we should do to
a. Hydra b. QN development process c. QN deployment process d. QN debugging process e. QN infrastructure routines
What we are looking for is to identify the really high value and cheap interventions we can do right now, because there are other urgent work we need to commence work on imminently, specifically
Deliverables Deadline
Submit as reply to this issue no later than:
Friday 6th 18pm CETDeliverables
For each person in the meeting , please review your past tasks, bugs, bug fixes, deployment problems, pain dealing with QN, and which of the buckets 1-5 it falls into. In particular perhaps
Agenda