Researching the queueing/ticketing system in the GCP context and in Cloud Run Services

[marcocastignoli]

Context

• Sourcify server struggles when many requests are incoming, it doesn't have any scaling strategy.
• Sourcify server currently have one pending HTTP request for each pending verification, /verify HTTP requests are closed only when verification is completed. With #1367 we want to introduce e receipt property in /v2/verify response, this will allow to separate HTTP request from the verification status.

Solutions

We are exploring two solutions:

• One that uses a queuing system between two new services: sourcify-http-server and sourcify-verification-service. sourcify-http-server will push pending contracts to the queue-service and sourcify-verification-service will read pending contracts from queue-service verifying them and marking them as completed. This solution involves setting up a queue service adding more complexity to our architecture, but we have granular control on what's in the queue, enabling us to potentially implement priority systems.

graph TD
    User -->|/verify| sourcify-http-server
    sourcify-http-server -->|Push Pending Contracts| queue-service
    queue-service -->|Read Pending Contracts| sourcify-verification-service
    sourcify-verification-service -->|Mark as Completed| sourcify-database
    sourcify-http-server -->|Read Status| sourcify-database
    sourcify-http-server -->|Read Status| queue-service

• The second solution doesn't use a queue service, but it takes advantage of the scaling solutions offered by GCP. Two new services sourcify-http-server and sourcify-verification-service will be deployed as Google Cloud Run Services, sourcify-http-server will receive /verify request from the internet and call sourcify-verification-service directly without passing through a queue. The verification's status is going to be saved in sourcify-database

graph TD
    User -->|/public_api_verify| sourcify-http-server
    sourcify-http-server -->|/internal_api_verify| sourcify-verification-service
    sourcify-verification-service -->|Write Status| sourcify-database
    sourcify-http-server -->|Read Status| sourcify-database

[kuzdogan]

• Do we need "granular control on what's in the queue" or "priority systems"?
• How do we send "tickets" in the second case?

[manuelwedler]

Could you explain what benefits the second proposal would bring compared to our current setup? Wouldn't just one server also scale well with GCP?

[marcocastignoli]

@manuelwedler

Could you explain what benefits the second proposal would bring compared to our current setup? Wouldn't just one server also scale well with GCP?

With our current setup we cannot unbind requests from verifications: a request is pending until the verification process is over. We need some way to separate verification from http requests if we want to support receipts in API v2

We could potentially separate them in the same "sourcify-server" process but then we are not taking advantage of GCP scaling by optimizing on the number of requests:

• cloud run would scale based on the number of opened http requests, but with this new hypothetical setup they are not related with verifications. So it will not optimize by number of pending verifications, but only by number of http requests.
• cloud run will also scale based on the amount of resources available, so it would actually scale, but not in an optimal way.

[marcocastignoli]

@kuzdogan

Do we need "granular control on what's in the queue" or "priority systems"?

I cannot think of any real use case of this, other than prioritizing some chains

How do we send "tickets" in the second case?

In the diagram I wrote "Read Status" from sourcify-http-server to sourcify-database:

• sourcify-http-server calls sourcify-verification-service triggering a new verification
• sourcify-verification-service stores a new receipt in the database as "pending" and returns the receipt id to sourcify-http-server, then starts the verification process, then marks the receipt as completed once done
• sourcify-http-server can always read the status reading directly from the database