Open gaius-qi opened 11 months ago
The scheduler should notify the peer to clear the cache and prevent the peer from hitting the local cache.
Currently Manager download will dispatch the download task to the peer node through scheduler, but it may not be expected on the peer download action, i.e., download failure, version error, etc., which leads to the download action on the peer node may have unexpected dirty cache. Currently in dfcache, it also provides access to a certain dfdaemon, and then operates to delete the task on the dfdaemon and reclaim the corresponding meta data and local storage. We need to provide a global api on the manager side to support asynchronous deletion of the cache corresponding to the task id of certain peers on the cluster, and to support getting the progress and tasks of the task.
The Manager provides a REST API to clear the Cache data for a P2P Task ID from all Peer nodes based on that Task ID. This requires the Manager to issue a move task to the Scheduler, and the Scheduler to call the GRPC interface of each Peer node containing that Task ID to clear the corresponding Cache. The cache clean supports clearing the cache that has been downloaded by peers on a d7y cluster. Users can call the manager api to operate the cluster cache clean task, and can view the progress of the task. The complete request timing diagram is as follows ListTasksJob:
DeleteTasksJob: Given task id, initiates a job that deletes the task cache on the peer.
When manager starts, it registers a REST API service on 8080 by default, which internally contains a number of API methods exposed on manager at manager/router/router.go
. Add a POST method to start an asynchronous task clearing the cache of the task id on peer, and a query to query the status of this asynchronous task and aggregate it into the creation logic of an existing asynchronous task.
The manager eventually collects the information needed to start the job and creates a GroupJob
into the machine to send a message to the distributed task waiting to be consumed by the scheduler.
There are existing job definitions that can reuse the job request to support new jobs.
// TODO Add auth to the following routes and fix the tests.
// Job.
job := apiv1.Group("/jobs")
job.POST("", h.CreateJob)
job.DELETE(":id", h.DestroyJob)
Support a new type job.
// internal/job/constants.go
// ListTasksJob is the name of listing tasks job.
ListTasksJob = "list_tasks"
// DeleteTasksJob is the name of deleting tasks job.
DeleteTaskJob = "delete_task"
Define a new JobRequest, response reuses the Job response, which
// For list task infos
type CreateListTasksJobRequest struct {
BIO string `json:"bio" binding:"omitempty"`
Type string `json:"type" binding:"required"`
Args ListTasksJobArgs `json:"args" binding:"omitempty"`
Result map[string]any `json:"result" binding:"omitempty"`
UserID uint `json:"user_id" binding:"omitempty"`
SchedulerClusterIDs []uint `json:"scheduler_cluster_ids" binding:"omitempty"`
}
type ListTasksJobArgs struct {
// TaskID is the task id for peer cache clean.
TaskID string `json:"taskID" binding:"required"`
}
// For delete tasks
type CreateDeleteTaskJobRequest struct {
// TODO: BIO define?
BIO string `json:"bio" binding:"omitempty"`
Type string `json:"type" binding:"required"`
Args DeleteTasksJob `json:"args" binding:"omitempty"`
Result map[string]any `json:"result" binding:"omitempty"`
UserID uint `json:"user_id" binding:"omitempty"`
SchedulerClusterIDs []uint `json:"scheduler_cluster_ids" binding:"omitempty"`
}
type DeleteTaskJob struct {
// TaskID is the task id for deleting task cache.
TaskID string `json:"taskID" binding:"required"`
}
In the return value definition, reuse the Job's definition manager/models/job.go
to populate the Resul
field with the task results corresponding to the above two jobs.
type Job struct {
BaseModel
...
Result JSONMap `gorm:"column:result;comment:group result" json:"result"`
...
}
For ListTasks
, we need to return a list of clusters/nodes that contain the taskid, for the sake of returning more sufficient information, we provide the original number of related Peers to be returned here.
// reference: manager/models/peer.go
type Peer struct {
BaseModel
Hostname string `gorm:"column:host_name;type:varchar(256);index:uk_peer,unique;not null;comment:hostname" json:"host_name"`
Type string `gorm:"column:type;type:varchar(256);index:idx_peer_type;comment:type" json:"type"`
IDC string `gorm:"column:idc;type:varchar(1024);comment:internet data center" json:"idc"`
Location string `gorm:"column:location;type:varchar(1024);comment:location" json:"location"`
IP string `gorm:"column:ip;type:varchar(256);index:uk_peer,unique;not null;comment:ip address" json:"ip"`
Port int32 `gorm:"column:port;not null;comment:grpc service listening port" json:"port"`
DownloadPort int32 `gorm:"column:download_port;not null;comment:download service listening port" json:"download_port"`
ObjectStoragePort int32 `gorm:"column:object_storage_port;comment:object storage service listening port" json:"object_storage_port"`
State string `gorm:"column:state;type:varchar(256);default:'inactive';comment:service state" json:"state"`
OS string `gorm:"column:os;type:varchar(256);comment:os" json:"os"`
Platform string `gorm:"column:platform;type:varchar(256);comment:platform" json:"platform"`
PlatformFamily string `gorm:"column:platform_family;type:varchar(256);comment:platform family" json:"platform_family"`
PlatformVersion string `gorm:"column:platform_version;type:varchar(256);comment:platform version" json:"platform_version"`
KernelVersion string `gorm:"column:kernel_version;type:varchar(256);comment:kernel version" json:"kernel_version"`
GitVersion string `gorm:"column:git_version;type:varchar(256);index:idx_peer_git_version;comment:git version" json:"git_version"`
GitCommit string `gorm:"column:git_commit;type:varchar(256);index:idx_peer_git_commit;comment:git commit" json:"git_commit"`
BuildPlatform string `gorm:"column:build_platform;type:varchar(256);comment:build platform" json:"build_platform"`
SchedulerClusterID uint `gorm:"index:uk_peer,unique;not null;comment:scheduler cluster id" json:"scheduler_cluster_id"`
SchedulerCluster SchedulerCluster `json:"scheduler_cluster"`
}
type ListTasksResponse struct {
Peers []Peer `json:"peers"`
}
For DeleteTask
, we need to return a list of success and failure nodes, paged at the api level.
// Reference: scheduler/storage/types.go
// Task defines the structure for task management.
type Task struct {
ID string `json:"id"`
PieceLength int `json:"piece_length"`
ContentLength int `json:"content_length"`
UploadingCount int `json:"uploading_count"`
UploadedCount int `json:"uploaded_count"`
UpdatedAt int64 `json:"updated_at"`
CreatedAt int64 `json:"created_at"`
PrefetchedAt int64 `json:"prefetched_at"`
FailedAt int64 `json:"failed_at"`
FinishedAt int64 `json:"finished_at"`
}
// TaskInfo includes information about a task along with peer details and a description.
type TaskInfo struct {
Task Task `json:"task"`
Peer Peer `json:"peer"`
Desc string `json:"desc"`
}
// DeleteTaskResponse represents the response after attempting to delete tasks,
// categorizing them into successfully and unsuccessfully deleted.
type DeleteTaskResponse struct {
SuccessTasks []TaskInfo `json:"success_tasks"`
FailureTasks []TaskInfo `json:"failure_tasks"`
}
Second, define interfaces in service manager/service/service.go
to support calls to lower level business logic.
// manager/service/service.go
CreateListTasksJob(context.Context, types.CreateListTasksJobRequest) (*models.Job, error)
CreateDeleteTaskJob(context.Context, types.CreateDeleteTasksJobRequest) (*models.Job, error)
CreatePeerCacheCleanJob
creates a Job
type via manager and creates a DeleteTask
and ListTasks
task via SendGroupWithContext
on the distributed queue, i.e., starts an asynchronous in redis task in redis, waiting for the scheduler to consume this task and execute it.
// manager/job/job.go
type Job struct {
*internaljob.Job
...
ListTasks
DeleteTasks
}
Scheduler starts a redis backend as a broker via machinery, RegisterJob
way to register that clean cache task to internaljob. Finally the function is executed to get the result
Define a job function in scheduler/job/job.go
:
// scheduler/job/job.go
// listTasks is a job to list tasks.
func (j *job) listTasks(ctx context.Context, data string) error {
// TODO:
// 1. query all peers with task id
return nil
}
// deleteTasks is a job to delete dirty tasks.
func (j *job) deleteTasks(ctx context.Context, data string) error {
// TODO:
// 1. query all peers with task id
// 2. delete task on all peers
return nil
}
In deleteTasks
, it tries to retrieve all peer ids that contain the task id and sends a delete task request to all peer ids (host) and returns the status of the task.
- Multiple batch calls to lower level api when calling interface
On the specific Peer side, the rpc request for delete task of scheduler will be executed (you can consider synchronous/asynchronous deletion), the main tasks to be done on the peer side are:
After defining proto, there are interfaces for defining delete task in dfdaemon(rust client), the core of which is the interface defined below:
// DfdaemonDownload represents download service of dfdaemon.
service DfdaemonDownload{
...
// DeleteTask deletes task from p2p network.
rpc DeleteTask(DeleteTaskRequest) returns(google.protobuf.Empty);
}
There is no implementation in the impl for the time being. Therefore, you can refer to the semantics of the go end and its own peer clean task to complete, specific to this implementation, the function needs to complete the work can refer to the following implementation:
// dragonfly-client/src/grpc/dfdaemon_download.rs
// delete_task calls the dfdaemon to delete the task.
#[instrument(skip_all)]
async fn delete_task(
&self,
request: Request<DeleteTaskRequest>,
) -> Result<Response<()>, Status> {
println!("delete_task: {:?}", request);
// TODO:
// 1. Delete meta data
// 2. Reclaim local storage
Err(Status::unimplemented("not implemented"))
}
Deleting metadata and taskdir from the local filesystem is already implemented in storage, which is kind of simplified compared to the go side, i.e. no marking of reclaim and direct deletion of the whole task directory.
In addition, the delete task here only cleans up in rocksdb, and the metadata is updated twice in dragon-client/src/task/mod.rs
, so if you clean up the metadata in the delete task, the download task overwrites the contents of the delete task.
Currently, meta data is directly written to the db, if you need to support delete behaviour, you need to judge whether the put operation is existed or not after the first put operation is started, if it is not existed, then it means it is deleted, and delete the dirty content generated by the current operation.
Another thing is that delete task needs to support announce task, there are GCs in rust that do delete task, you should have to aggregate announce task (or should be called boardcast) into delete_task implementation within Storage.
// dragonfly-client-storage/src/lib.rs
// delete_task deletes the task metadatas, task content and piece metadatas.
pub async fn delete_task(&self, id: &str) -> Result<()> {
self.metadata.delete_task(id)?;
self.metadata.delete_pieces(id)?;
self.content.delete_task(id).await?;
Ok(())
}
Align unit test with other jobs in manager and scheduler, add mock job type:
// manager/handlers/job_test.go
var (
mockDeleteTasksReqBody = ...
...
)
Define the DeleteTasks and ListTasks mocks in manager/job/mocks
, refer to manager/job/mocks/preheat_mock.go
.
There is no need to add a new unit test in the scheduler for now.
Referring to preheat's e2e test, test/e2e/v2/manager/preheat.go
, define the test case for the ginkgo test, which needs to include the query and delete It() test points.
url: /jobs request/response body:
list task request
{
"args": {
"task_id": "string",
"page": "int",
"per_page": "int",
},
"bio": "",
"result": {
},
"scheduler_cluster_ids": [
0
],
"seed_peer_cluster_ids": [
0
],
"type": "list_tasks",
"user_id": 0
}
list task response
{
"args": {
"task_id": "string",
"page": "int",
"per_page": "int",
},
"bio": "string",
"created_at": "string",
"id": 0,
"is_del": 0,
"result": {
},
"scheduler_clusters": [
{
"bio": "string",
"client_config": {
},
"config": {
},
"created_at": "string",
"id": 0,
"is_default": true,
"is_del": 0,
"jobs": [
"string"
],
"name": "string",
"peers": [
{
"build_platform": "string",
"created_at": "string",
"download_port": 0,
"git_commit": "string",
"git_version": "string",
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"kernel_version": "string",
"location": "string",
"object_storage_port": 0,
"os": "string",
"platform": "string",
"platform_family": "string",
"platform_version": "string",
"port": 0,
"scheduler_cluster": "string",
"scheduler_cluster_id": 0,
"state": "string",
"type": "string",
"updated_at": "string"
}
],
"schedulers": [
{
"created_at": "string",
"features": [
"string"
],
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"location": "string",
"models": [
{
"bio": "string",
"created_at": "string",
"evaluation": {
},
"id": 0,
"is_del": 0,
"name": "string",
"scheduler": "string",
"scheduler_id": 0,
"state": "string",
"type": "string",
"updated_at": "string",
"version": "string"
}
],
"port": 0,
"scheduler_cluster": "string",
"scheduler_cluster_id": 0,
"state": "string",
"updated_at": "string"
}
],
"scopes": {
},
"seed_peer_clusters": [
{
"bio": "string",
"config": {
},
"created_at": "string",
"id": 0,
"is_del": 0,
"jobs": [
"string"
],
"name": "string",
"scheduler_clusters": [
"string"
],
"seed_peer": [
{
"created_at": "string",
"download_port": 0,
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"location": "string",
"object_storage_port": 0,
"port": 0,
"seed_peer_cluster": "string",
"seed_peer_cluster_id": 0,
"state": "string",
"type": "string",
"updated_at": "string"
}
],
"updated_at": "string"
}
],
"updated_at": "string"
}
],
"seed_peer_clusters": [
{
"bio": "string",
"config": {
},
"created_at": "string",
"id": 0,
"is_del": 0,
"jobs": [
"string"
],
"name": "string",
"scheduler_clusters": [
"string"
],
"seed_peer": [
{
"created_at": "string",
"download_port": 0,
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"location": "string",
"object_storage_port": 0,
"port": 0,
"seed_peer_cluster": "string",
"seed_peer_cluster_id": 0,
"state": "string",
"type": "string",
"updated_at": "string"
}
],
"updated_at": "string"
}
],
"state": "string",
"task_id": "string",
"type": "string",
"updated_at": "string",
"user": {
"avatar": "string",
"bio": "string",
"configs": [
{
"bio": "string",
"created_at": "string",
"id": 0,
"is_del": 0,
"name": "string",
"updated_at": "string",
"user": "string",
"user_id": 0,
"value": "string"
}
],
"created_at": "string",
"email": "string",
"id": 0,
"is_del": 0,
"location": "string",
"name": "string",
"phone": "string",
"state": "string",
"updated_at": "string"
},
"user_id": 0
}
delete task request
{
"args": {
"task_id": "string",
},
"bio": "string",
"result": {
},
"scheduler_cluster_ids": [
0
],
"seed_peer_cluster_ids": [
0
],
"type": "delete_task",
"user_id": 0
}
delete task response
{
"args": {
"task_id": "string",
},
"bio": "string",
"created_at": "string",
"id": 0,
"is_del": 0,
"result": {
},
"scheduler_clusters": [
{
"bio": "string",
"client_config": {
},
"config": {
},
"created_at": "string",
"id": 0,
"is_default": true,
"is_del": 0,
"jobs": [
"string"
],
"name": "string",
"peers": [
{
"build_platform": "string",
"created_at": "string",
"download_port": 0,
"git_commit": "string",
"git_version": "string",
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"kernel_version": "string",
"location": "string",
"object_storage_port": 0,
"os": "string",
"platform": "string",
"platform_family": "string",
"platform_version": "string",
"port": 0,
"scheduler_cluster": "string",
"scheduler_cluster_id": 0,
"state": "string",
"type": "string",
"updated_at": "string"
}
],
"schedulers": [
{
"created_at": "string",
"features": [
"string"
],
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"location": "string",
"models": [
{
"bio": "string",
"created_at": "string",
"evaluation": {
},
"id": 0,
"is_del": 0,
"name": "string",
"scheduler": "string",
"scheduler_id": 0,
"state": "string",
"type": "string",
"updated_at": "string",
"version": "string"
}
],
"port": 0,
"scheduler_cluster": "string",
"scheduler_cluster_id": 0,
"state": "string",
"updated_at": "string"
}
],
"scopes": {
},
"seed_peer_clusters": [
{
"bio": "string",
"config": {
},
"created_at": "string",
"id": 0,
"is_del": 0,
"jobs": [
"string"
],
"name": "string",
"scheduler_clusters": [
"string"
],
"seed_peer": [
{
"created_at": "string",
"download_port": 0,
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"location": "string",
"object_storage_port": 0,
"port": 0,
"seed_peer_cluster": "string",
"seed_peer_cluster_id": 0,
"state": "string",
"type": "string",
"updated_at": "string"
}
],
"updated_at": "string"
}
],
"updated_at": "string"
}
],
"seed_peer_clusters": [
{
"bio": "string",
"config": {
},
"created_at": "string",
"id": 0,
"is_del": 0,
"jobs": [
"string"
],
"name": "string",
"scheduler_clusters": [
"string"
],
"seed_peer": [
{
"created_at": "string",
"download_port": 0,
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"location": "string",
"object_storage_port": 0,
"port": 0,
"seed_peer_cluster": "string",
"seed_peer_cluster_id": 0,
"state": "string",
"type": "string",
"updated_at": "string"
}
],
"updated_at": "string"
}
],
"state": "string",
"task_id": "string",
"type": "string",
"updated_at": "string",
"user": {
"avatar": "string",
"bio": "string",
"configs": [
{
"bio": "string",
"created_at": "string",
"id": 0,
"is_del": 0,
"name": "string",
"updated_at": "string",
"user": "string",
"user_id": 0,
"value": "string"
}
],
"created_at": "string",
"email": "string",
"id": 0,
"is_del": 0,
"location": "string",
"name": "string",
"phone": "string",
"state": "string",
"updated_at": "string"
},
"user_id": 0
}
url: /jobs/{id} response body:
list task request response,the response data is in result
property.
{
"args": {
"task_id": "string",
"page": "int",
"per_page": "int",
},
"bio": "string",
"created_at": "string",
"id": 0,
"is_del": 0,
"result": {
"peers": [
{
"host_name": "server1.example.com",
"type": "primary",
"idc": "IDC1",
"location": "New York, NY",
"ip": "192.168.1.1",
"port": 8080,
"download_port": 9090,
"object_storage_port": 10000,
"state": "active",
"os": "Linux",
"platform": "x86_64",
"platform_family": "debian",
"platform_version": "10",
"kernel_version": "4.19.0-12-amd64",
"git_version": "v1.2.3",
"git_commit": "a1b2c3d4",
"build_platform": "linux/amd64",
"scheduler_cluster_id": 1,
"scheduler_cluster": {
"id": 1,
"name": "Cluster A",
"state": "active",
"region": "us-east-1"
}
},
{
"host_name": "server2.example.com",
"type": "secondary",
"idc": "IDC2",
"location": "San Francisco, CA",
"ip": "192.168.1.2",
"port": 8081,
"download_port": 9091,
"object_storage_port": 10001,
"state": "inactive",
"os": "Linux",
"platform": "x86_64",
"platform_family": "ubuntu",
"platform_version": "18.04",
"kernel_version": "5.4.0-42-generic",
"git_version": "v1.2.4",
"git_commit": "e5f6g7h8",
"build_platform": "linux/amd64",
"scheduler_cluster_id": 2,
"scheduler_cluster": {
"id": 2,
"name": "Cluster B",
"state": "inactive",
"region": "us-west-2"
}
}
]
},
"scheduler_clusters": [
{
"bio": "string",
"client_config": {
},
"config": {
},
"created_at": "string",
"id": 0,
"is_default": true,
"is_del": 0,
"jobs": [
"string"
],
"name": "string",
"peers": [
{
"build_platform": "string",
"created_at": "string",
"download_port": 0,
"git_commit": "string",
"git_version": "string",
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"kernel_version": "string",
"location": "string",
"object_storage_port": 0,
"os": "string",
"platform": "string",
"platform_family": "string",
"platform_version": "string",
"port": 0,
"scheduler_cluster": "string",
"scheduler_cluster_id": 0,
"state": "string",
"type": "string",
"updated_at": "string"
}
],
"schedulers": [
{
"created_at": "string",
"features": [
"string"
],
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"location": "string",
"models": [
{
"bio": "string",
"created_at": "string",
"evaluation": {
},
"id": 0,
"is_del": 0,
"name": "string",
"scheduler": "string",
"scheduler_id": 0,
"state": "string",
"type": "string",
"updated_at": "string",
"version": "string"
}
],
"port": 0,
"scheduler_cluster": "string",
"scheduler_cluster_id": 0,
"state": "string",
"updated_at": "string"
}
],
"scopes": {
},
"seed_peer_clusters": [
{
"bio": "string",
"config": {
},
"created_at": "string",
"id": 0,
"is_del": 0,
"jobs": [
"string"
],
"name": "string",
"scheduler_clusters": [
"string"
],
"seed_peer": [
{
"created_at": "string",
"download_port": 0,
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"location": "string",
"object_storage_port": 0,
"port": 0,
"seed_peer_cluster": "string",
"seed_peer_cluster_id": 0,
"state": "string",
"type": "string",
"updated_at": "string"
}
],
"updated_at": "string"
}
],
"updated_at": "string"
}
],
"seed_peer_clusters": [
{
"bio": "string",
"config": {
},
"created_at": "string",
"id": 0,
"is_del": 0,
"jobs": [
"string"
],
"name": "string",
"scheduler_clusters": [
"string"
],
"seed_peer": [
{
"created_at": "string",
"download_port": 0,
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"location": "string",
"object_storage_port": 0,
"port": 0,
"seed_peer_cluster": "string",
"seed_peer_cluster_id": 0,
"state": "string",
"type": "string",
"updated_at": "string"
}
],
"updated_at": "string"
}
],
"state": "string",
"task_id": "string",
"type": "string",
"updated_at": "string",
"user": {
"avatar": "string",
"bio": "string",
"configs": [
{
"bio": "string",
"created_at": "string",
"id": 0,
"is_del": 0,
"name": "string",
"updated_at": "string",
"user": "string",
"user_id": 0,
"value": "string"
}
],
"created_at": "string",
"email": "string",
"id": 0,
"is_del": 0,
"location": "string",
"name": "string",
"phone": "string",
"state": "string",
"updated_at": "string"
},
"user_id": 0
}
delete task response, the response is in result
property
{
"args": {
"task_id": "string",
},
"bio": "string",
"created_at": "string",
"id": 0,
"is_del": 0,
"result": {
"success_tasks": [
{
"task": {
"id": "task1",
"piece_length": 1024,
"content_length": 20480,
"uploading_count": 5,
"uploaded_count": 5,
"updated_at": 1599557760,
"created_at": 1599554160,
"prefetched_at": 1599554460,
"failed_at": 0,
"finished_at": 1599557760
},
"peer": {
"id": "peer1",
"address": "192.168.1.100",
"port": 8080
},
"desc": "Task completed successfully."
},
{
"task": {
"id": "task2",
"piece_length": 2048,
"content_length": 40960,
"uploading_count": 3,
"uploaded_count": 3,
"updated_at": 1599657760,
"created_at": 1599654160,
"prefetched_at": 1599654460,
"failed_at": 0,
"finished_at": 1599657760
},
"peer": {
"id": "peer2",
"address": "192.168.1.101",
"port": 8081
},
"desc": "Task completed without issues."
}
],
"failure_tasks": [
{
"task": {
"id": "task3",
"piece_length": 512,
"content_length": 10240,
"uploading_count": 2,
"uploaded_count": 1,
"updated_at": 1599757760,
"created_at": 1599754160,
"prefetched_at": 0,
"failed_at": 1599755000,
"finished_at": 0
},
"peer": {
"id": "peer3",
"address": "192.168.1.102",
"port": 8082
},
"desc": "Task failed due to network issues."
}
]
},
"scheduler_clusters": [
{
"bio": "string",
"client_config": {
},
"config": {
},
"created_at": "string",
"id": 0,
"is_default": true,
"is_del": 0,
"jobs": [
"string"
],
"name": "string",
"peers": [
{
"build_platform": "string",
"created_at": "string",
"download_port": 0,
"git_commit": "string",
"git_version": "string",
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"kernel_version": "string",
"location": "string",
"object_storage_port": 0,
"os": "string",
"platform": "string",
"platform_family": "string",
"platform_version": "string",
"port": 0,
"scheduler_cluster": "string",
"scheduler_cluster_id": 0,
"state": "string",
"type": "string",
"updated_at": "string"
}
],
"schedulers": [
{
"created_at": "string",
"features": [
"string"
],
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"location": "string",
"models": [
{
"bio": "string",
"created_at": "string",
"evaluation": {
},
"id": 0,
"is_del": 0,
"name": "string",
"scheduler": "string",
"scheduler_id": 0,
"state": "string",
"type": "string",
"updated_at": "string",
"version": "string"
}
],
"port": 0,
"scheduler_cluster": "string",
"scheduler_cluster_id": 0,
"state": "string",
"updated_at": "string"
}
],
"scopes": {
},
"seed_peer_clusters": [
{
"bio": "string",
"config": {
},
"created_at": "string",
"id": 0,
"is_del": 0,
"jobs": [
"string"
],
"name": "string",
"scheduler_clusters": [
"string"
],
"seed_peer": [
{
"created_at": "string",
"download_port": 0,
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"location": "string",
"object_storage_port": 0,
"port": 0,
"seed_peer_cluster": "string",
"seed_peer_cluster_id": 0,
"state": "string",
"type": "string",
"updated_at": "string"
}
],
"updated_at": "string"
}
],
"updated_at": "string"
}
],
"seed_peer_clusters": [
{
"bio": "string",
"config": {
},
"created_at": "string",
"id": 0,
"is_del": 0,
"jobs": [
"string"
],
"name": "string",
"scheduler_clusters": [
"string"
],
"seed_peer": [
{
"created_at": "string",
"download_port": 0,
"host_name": "string",
"id": 0,
"idc": "string",
"ip": "string",
"is_del": 0,
"location": "string",
"object_storage_port": 0,
"port": 0,
"seed_peer_cluster": "string",
"seed_peer_cluster_id": 0,
"state": "string",
"type": "string",
"updated_at": "string"
}
],
"updated_at": "string"
}
],
"state": "string",
"task_id": "string",
"type": "string",
"updated_at": "string",
"user": {
"avatar": "string",
"bio": "string",
"configs": [
{
"bio": "string",
"created_at": "string",
"id": 0,
"is_del": 0,
"name": "string",
"updated_at": "string",
"user": "string",
"user_id": 0,
"value": "string"
}
],
"created_at": "string",
"email": "string",
"id": 0,
"is_del": 0,
"location": "string",
"name": "string",
"phone": "string",
"state": "string",
"updated_at": "string"
},
"user_id": 0
}
url: /jobs request/response body:
{
"args": {
"task_id": "string",
},
"type": "get_task",
}
{
"args": {
"task_id": "string",
},
"bio": "string",
"created_at": "string",
"id": 0,
"is_del": 0,
"result": {
},
"scheduler_clusters": [
],
"seed_peer_clusters": [
],
"state": "string",
"task_id": "string",
"type": "string",
"updated_at": "string",
"user": {
},
"user_id": 0
}
{
"args": {
"task_id": "string",
},
"type": "delete_task",
}
{
"args": {
"task_id": "string",
},
"bio": "string",
"created_at": "string",
"id": 0,
"is_del": 0,
"result": {
},
"scheduler_clusters": [
],
"seed_peer_clusters": [
],
"state": "string",
"task_id": "string",
"type": "string",
"updated_at": "string",
"user": {
},
"user_id": 0
}
url: /jobs/{id} response body:
result
property.
{
"args": {
"task_id": "string",
},
"bio": "string",
"created_at": "string",
"id": 0,
"is_del": 0,
"result": {
"peers": [
{
"host_name": "server1.example.com",
"type": "primary",
"idc": "IDC1",
"location": "New York, NY",
"ip": "192.168.1.1",
"port": 8080,
"download_port": 9090,
"object_storage_port": 10000,
"state": "active",
"os": "Linux",
"platform": "x86_64",
"platform_family": "debian",
"platform_version": "10",
"kernel_version": "4.19.0-12-amd64",
"git_version": "v1.2.3",
"git_commit": "a1b2c3d4",
"build_platform": "linux/amd64",
"scheduler_cluster_id": 1,
"scheduler_cluster": {
"id": 1,
"name": "Cluster A",
"state": "active",
"region": "us-east-1"
}
},
{
"host_name": "server2.example.com",
"type": "secondary",
"idc": "IDC2",
"location": "San Francisco, CA",
"ip": "192.168.1.2",
"port": 8081,
"download_port": 9091,
"object_storage_port": 10001,
"state": "inactive",
"os": "Linux",
"platform": "x86_64",
"platform_family": "ubuntu",
"platform_version": "18.04",
"kernel_version": "5.4.0-42-generic",
"git_version": "v1.2.4",
"git_commit": "e5f6g7h8",
"build_platform": "linux/amd64",
"scheduler_cluster_id": 2,
"scheduler_cluster": {
"id": 2,
"name": "Cluster B",
"state": "inactive",
"region": "us-west-2"
}
}
],
},
"scheduler_clusters": [
],
"seed_peer_clusters": [
],
"state": "string",
"task_id": "string",
"type": "get_task",
"updated_at": "string",
"user": {
},
"user_id": 0
}
result
property{
"args": {
"task_id": "string"
},
"bio": "string",
"created_at": "string",
"id": 0,
"is_del": 0,
"result": {
"success_peers": [
{
"peer": {
"host_name": "server2.example.com",
"type": "secondary",
"idc": "IDC2",
"location": "San Francisco, CA",
"ip": "192.168.1.2",
"port": 8081,
"download_port": 9091,
"object_storage_port": 10001,
"state": "inactive",
"os": "Linux",
"platform": "x86_64",
"platform_family": "ubuntu",
"platform_version": "18.04",
"kernel_version": "5.4.0-42-generic",
"git_version": "v1.2.4",
"git_commit": "e5f6g7h8",
"build_platform": "linux/amd64",
"scheduler_cluster_id": 2,
"scheduler_cluster": {
"id": 2,
"name": "Cluster B",
"state": "inactive",
"region": "us-west-2"
}
},
"description": ""
}
],
"failure_peers": [
{
"peer": {
"host_name": "server2.example.com",
"type": "secondary",
"idc": "IDC2",
"location": "San Francisco, CA",
"ip": "192.168.1.2",
"port": 8081,
"download_port": 9091,
"object_storage_port": 10001,
"state": "inactive",
"os": "Linux",
"platform": "x86_64",
"platform_family": "ubuntu",
"platform_version": "18.04",
"kernel_version": "5.4.0-42-generic",
"git_version": "v1.2.4",
"git_commit": "e5f6g7h8",
"build_platform": "linux/amd64",
"scheduler_cluster_id": 2,
"scheduler_cluster": {
"id": 2,
"name": "Cluster B",
"state": "inactive",
"region": "us-west-2"
}
},
"description": "Error message."
}
],
"scheduler_clusters": [
],
"seed_peer_clusters": [
],
"state": "string",
"task_id": "string",
"type": "delete_task",
"updated_at": "string",
"user": {
},
"user_id": 0
}
Feature request:
Use case:
UI Example: