plebhash commented 3 months ago

Originally written by @rrybarczyk on November 15th 2022

SRI cgminer SV1/SV2 Translator Application

Proposed Firmware MVP for SRI

SV1/SV2 protocol translator logic (written in Rust) will be added to the frontend of cgminer (written in C). The translator logic will use SRI’s sv2-core library and will communicate to via language bindings. The translator logic and bindings will be compiled in the same binary as cgminer, but the core of cgminer will be preserved. The result will give the firmware user the ability to communicate with SV2 pools and also maintain its backward compatibility with SV1 pools.

Background

The SRI team is looking to support firmware applications. There are several ways to provide downstream firmware application support–the most obvious being pure SV2 firmware written from scratch in Rust using SRI’s sv2-core libraries. However, this is may not be the best approach when factoring in adoption time, which is of high importance to the project.

It is likely that the fastest path to ubiquitous SV2 adoption is getting hardware manufacturer’s on board to shipping SV2-compatible firmware as their OEM. When discussing the transition to SV2 with hardware manufacturers, their biggest concern is introducing new bugs into their firmware. Because of this, there is is value in creating an application that preserves as much logic as possible while still resulting in a firmware application with SV2 support. This application should be viewed as a medium-term stepping stone for future, more comprehensive SRI firmware applications.

Currently, manufacturer use an internal fork of the open source cgminer main branch, which supports the SV1 protocol only. The application will also be open source and will be available for any manufacturer or individual to pull into their cgminer firmware, allowing them to support both the SV1 and SV2 protocols. This application will contain SV1/SV2 translation logic, interfacing with the most external cgminer functions (perserving cgminer completely), and connecting to a SV2 upstream role (pool or proxy). Cmginer is written in C, but this application will be written in Rust and will interface with cgminer via a language bindings.

Components Overview

cgminer

The largest mining hardware manufacturers use an internal fork of cgminer in the application layer of their OEM firmware which is only compatible with the SV1 protocol. Cgminer is written in C.

It is important to note that the application to be written is hardware agnostic, therefore no one mining device model or manufacturer is directly favored. To develop the application, a block-eruptor USB miner will be used, which cgminer has driver support for.

The highlevel development configuration is shown in Figure 1 below.

Figure 1: Highlevel Architecture Overview

The backend of cgminer is the variable part that is use-case/hardware specific (usb-miner, S9, whatsminer, etc.). The frontend of cgminer is the generic part that is common to all cgminer implementations: it has the SV1 protocol implementation, job validation, etc. Note that the front end and back end of cgminer is packaged in the same binary, and is simply split in the diagram to help ease the understanding of where the logic for the two options will live.

SRI sv2-core

SRI’s sv2-core library is an implementation of the SV2 protocol that can be used to create a variety of applications, including both Options explored in this proposal. Sv2-core is written in Rust.

Firmware Translator Application Overview

A firmware translation application that interfaces with cgminer is likely the fastest way to get manufacturer’s to ship SV2 by default on their machines because no cgminer modification is required. This involves creating new SV1/SV2 translation logic to be included in cgminer’s frontend. This logic will have an upstream connection with a SV2 role (pool or proxy) and a downstream connection to six of cgminer’s most external facing functions. These six functions are in cgminer’s util.c file and are the following:

The new translation logic will send and receive SV2 messages to its upstream connection as well as send and receive SV1 messages to one of these six cgminer functions, performing the proper protocol translation where necessary. This new translation will be an addition to cgminer’s frontend only, preserving all cgminer logic as is.

The highlevel development configuration is shown in Figure 2 below.

Figure 2: Firmware SV1/SV2 Translation Application High level Architecture Overview

From cgminer’s perspective, there is no change to its functionality. The only change is instead of interfacing directly with an upstream SV1 role, the six functions in util.c listed above communicate with the new translation logic (via language bindings). No modification is made to the cgminer code (frontend or backend) beyond adding in the translation logic.

This preservation of cgminer is attractive to manufacturers given their hesitation to modify their existing firmware for fear of introducing fatal bugs.

Pros

Easiest way in terms of both dev time and implementation complexity to make cgminer SV2-compatible.
With the information that we have available now, this is likely the fastest path to get SV2 adopted by the hardware manufacterures.

Cons

Cgminer is still in use, so all of its problems are still inherited. Some of these problems are related to how giant and unmaintained the code is, which can result in inefficiencies and bugs. Other problems relate to potential backdoors that manufacturers may have included. However, the only way to truly get around this is to completely replace cgminer with an open-source rewrite which all manufacturers agree to use. As manufacturers are very protective over their firmware and keep it locked down, the use of an auditable, open source implementation as the OEM will be a long term battle.

Alternative Solutions

Build the SV1/SV2 translator logic in C to remove the need for language bindings.
- Not chosen because SRI’s sv2-core library is a heavy dependency for the translator logic that would be more work to rewrite in C
Rewrite all of cgminer in Rust, starting at cgminer’s backend driver interface.
- This would be a more sustainable, long term solution, but it requires significantly more developer work.
- This may deter use as some manufacturers may be hesitant to use firmware that deviates significantly from their existing cgminer firmware for fear of introducing new bugs.

plebhash commented 3 months ago

I started implementing this in a slightly different direction.

Instead of leveraging low-level crates from SRI, I'm building an entire Translator Proxy to be spawned inside cgminer via pthread_create. cgminer can connect to it internally as if it were a pool.

plebhash commented 1 month ago

cgminer CLI is highly ossified towards SV1 use-cases, and the codebase is confusing and hard to navigate.

We should consider also refactoring the CLI in Rust.

plebhash commented 1 month ago

I started implementing this in a slightly different direction.

Instead of leveraging low-level crates from SRI, I'm building an entire Translator Proxy to be spawned inside cgminer via pthread_create. cgminer can connect to it internally as if it were a pool.

having second thoughts about this approach... too hacky and dirty

rrybarczyk commented 3 weeks ago

I think the first approach here is to remove the difficulty aggregation from the translator proxy. Then we have a standalone application that can be used for anything, including firmware. @xraid may also have ideas here.

Also, we need to make sure we have really robust logging on the tproxy, especially for this purpose. This will help debug so we can see which templates we are getting, when we are getting them, ect. And also this will open up the ability to plug in a dashboard for a production application in the future.

We already use tracing for logging, which works great for this. It not only logs to files, but can send logs to a web socket (among other things). We really need to think about how we want to standardize our log messages, adding tags like tp or template or upstream, etc. Have not thought too much here.

xraid commented 3 weeks ago

There is some effort expressed to make embedded Rust SV2 for esp2 bitaxe nerdminer and other FW etc. = smallish MD's by NebulaMiner / GPTechinno( i invite him here)

A Cross compile of a T role (tProxy sans aggregation) to Arm9 = Antminer Xilinx controlboards, will result in be able run T-app on miner direct and just config miner to localhost:9999 where the T-app points to a SV2 endpoint.

Moving below to separate issue https://github.com/stratum-mining/stratum/issues/998 ***

"rrybarczyk"xraid may also have ideas here."

I am proposing a dialog of SRI-RT (RT = runtime) to offload roles while letting roles just do Tracing-log-file and have RT do filter and massage / presentation of throughput.

Tracing log to rolling file can be used for unified debug of roles in same UI table (sort / search), and also do health, stats, monitoring + do the accounting for role,.

This would let SRI-Roles all be leaner from extra add-on code, that in turn makes SRI-Roles code more readable and maintainable.

"Simplicity is very Complex" A new SRI-RT Role would add complexity to system while make it simpler to run and manage.

SRI-RT would help dload(install role), config, monitor and have accounting filtered to store. Also help in have a unified debug / health log UI view of connected roles.

Vision is install SRI-RT and from there a CLI / UI / API is used to dload, config, run roles. Where it also acts as janitor for the tracing-log for view etc.

SRI-RT is a different mental model of the SRI system, SRI-RT would be the system of workers(roles).

xraid commented 3 weeks ago

ok . NebulaMiner is at Mining Disrupt and will eventually chime in after ...

plebhash / cgminer-sv2