Comprehensive Guide to Building Exchange Connectors for Our Trading Infrastructure

Introduction

Welcome to the development team! This guide is designed to help you build public and private exchange connectors for our market-making infrastructure. Exchange connectors are critical components that interact with exchanges to retrieve market data and execute trades.

This guide will walk you through the general structure and steps needed to implement both public and private connectors, using provided code examples and incorporating best practices observed from existing implementations. As you develop the connectors, you should also consult the exchange-specific API documentation to handle any nuances or unique requirements. Please note that this guide is not a "one size fits all" approach. Each Exchnage might require very diffrent approaches.

---

Table of Contents

1. Overview of Exchange Connectors
   • Public Connectors
   • Private Connectors
2. Prerequisites
3. Common Components
4. Building a Public Connector
   • 1. Set Up the Connector Class
   • 2. Define Constructor and Member Variables
   • 3. Establish the WebSocket Connection
   • 4. Subscribe to Market Data Channels
   • 5. Handle Incoming Messages
   • 6. Implement Event Type Determination
   • 7. Create Serializable Events
   • 8. Handle Authentication (If Required)
   • 9. Implement Reconnection Logic
   • 10. Implement the stop Method
   • 11. Update the Main Public Connector
   • 12. Testing and Validation
5. Building a Private Connector
   • 1. Set Up the Connector Class
   • 2. Define Constructor and Member Variables
   • 3. Establish the WebSocket and REST Connections
   • 4. Handle Authentication
     • Handling Listen Keys or Session Tokens
   • 5. Subscribe to Private Channels
   • 6. Implement RPC Methods
     • Place Orders
     • Implementing Order Batching
     • Cancel Orders
     • Get Open Orders
     • Get Balance
   • 7. Handle Incoming Messages
   • 8. Implement Reconnection Logic
     • Ping Mechanism and Cleanup
   • 9. Implement the stop Method
   • 10. Update the Main Private Connector
   • 11. Testing and Validation
6. Best Practices
7. Conclusion
8. Appendix

---

1. Overview of Exchange Connectors

Exchange connectors are divided into two categories:

Public Connectors

• Purpose: Connect to an exchange's public market data feeds to receive real-time information such as order books, trades, and tickers.
• Responsibilities:
  • Establish WebSocket connections to public APIs.
  • Subscribe to market data channels.
  • Process and standardize incoming data.
  • Handle reconnection logic and error handling.

Private Connectors

• Purpose: Interact with an exchange's private APIs to execute trades, manage orders, and retrieve account information.
• Responsibilities:
  • Authenticate with the exchange using API keys and secrets.
  • Establish WebSocket and/or REST connections for private endpoints.
  • Implement methods to place orders, cancel orders, retrieve open orders, and get balances.
  • Handle order status updates and account events.
  • Manage reconnection and error handling.

---

2. Prerequisites

• Proficiency in TypeScript and Node.js.
• Understanding of WebSocket and RESTful APIs.
• Familiarity with asynchronous programming and event-driven architecture.
• Access to exchange-specific API documentation.
• Knowledge of our internal data types and interfaces from the skl-shared library.

---

3. Common Components

Both public and private connectors share some common elements:

• Connector Interfaces: Implement the appropriate interface (PublicExchangeConnector or PrivateExchangeConnector) to ensure consistency.
• Symbol and Side Mapping: Use utility functions to map exchange-specific symbols and sides to our internal representations.
• Event Handling: Process incoming messages and convert them to our standardized Serializable types.
• Error Handling and Logging: Implement comprehensive error handling and use the Logger utility for logging.
• Reconnection Logic: Ensure connectors can recover from disconnections automatically.

---

4. Building a Public Connector

1. Set Up the Connector Class

• Create a new class in src/connectors/public/.
• Implement the PublicExchangeConnector interface.

import {
    PublicExchangeConnector,
    ConnectorConfiguration,
    ConnectorGroup,
    Serializable,
} from 'skl-shared';

export class ExchangeNamePublicConnector implements PublicExchangeConnector {
    // Implementation
}

---

2. Define Constructor and Member Variables

• Constructor Parameters:

  • group: ConnectorGroup
  • config: ConnectorConfiguration
  • credential?: Credential (if authentication is required)

• Member Variables:

  • Exchange symbols (exchangeSymbol, sklSymbol)
  • WebSocket endpoints
  • WebSocket client instance
  • State variables (e.g., order book levels)

constructor(
    private group: ConnectorGroup,
    private config: ConnectorConfiguration,
    private credential?: Credential,
) {
    this.exchangeSymbol = getExchangeSymbol(this.group, this.config);
    this.sklSymbol = getSklSymbol(this.group, this.config);
}

---

3. Establish the WebSocket Connection

• Initialize the WebSocket client.
• Implement the connect method, which accepts a callback onMessage to handle incoming data.

public async connect(onMessage: (messages: Serializable[]) => void): Promise<void> {
    this.websocket = new WebSocket(this.publicWebsocketUrl);

    this.websocket.on('open', () => {
        this.subscribeToChannels();
    });

    this.websocket.on('message', (data: string) => {
        this.handleMessage(data, onMessage);
    });

    this.websocket.on('error', (error: Error) => {
        // Handle errors
    });

    this.websocket.on('close', () => {
        // Reconnect logic
    });
}

---

4. Subscribe to Market Data Channels

• Determine necessary channels (e.g., trades, order book updates, tickers).
• Format subscription messages according to the exchange's API specification.
• Send subscription messages upon connection.

private subscribeToChannels(): void {
    const channels = [
        `trades.${this.exchangeSymbol}`,
        `orderbook.${this.exchangeSymbol}`,
        `ticker.${this.exchangeSymbol}`,
    ];

    const subscriptionMessage = {
        method: 'SUBSCRIBE',
        params: channels,
        // Include authentication if required
    };

    this.websocket.send(JSON.stringify(subscriptionMessage));
}

---

5. Handle Incoming Messages

• Parse messages and determine event type.
• Convert data into Serializable types.
• Invoke onMessage callback.

private handleMessage(data: string, onMessage: (messages: Serializable[]) => void): void {
    const message = JSON.parse(data);
    const eventType = this.getEventType(message);

    if (eventType) {
        const serializableMessages = this.createSerializableEvents(eventType, message);
        if (serializableMessages.length > 0) {
            onMessage(serializableMessages);
        }
    } else {
        // Log unrecognized messages
    }
}

---

6. Implement Event Type Determination

• Map incoming messages to SklEvent types.
• Handle special cases or message formats.

private getEventType(message: any): SklEvent | null {
    if (message.type === 'trade') {
        return 'Trade';
    } else if (message.type === 'orderbook') {
        return 'TopOfBook';
    } else if (message.type === 'ticker') {
        return 'Ticker';
    } else if (message.error) {
        logger.error(`Error message received: ${message.error}`);
        return null;
    } else if (message.event === 'subscription') {
        logger.info(`Subscription confirmed: ${JSON.stringify(message)}`);
        return null;
    }
    return null;
}

---

7. Create Serializable Events

• Transform exchange data into internal structures.
• Use utility functions for symbol and side mappings.

private createSerializableEvents(eventType: SklEvent, message: any): Serializable[] {
    switch (eventType) {
        case 'Trade':
            return [this.createTrade(message)];
        case 'TopOfBook':
            return [this.createTopOfBook(message)];
        case 'Ticker':
            return [this.createTicker(message)];
        default:
            return [];
    }
}

private createTrade(message: any): Trade {
    return {
        symbol: this.sklSymbol,
        connectorType: 'ExchangeName',
        event: 'Trade',
        price: parseFloat(message.price),
        size: parseFloat(message.size),
        side: mapExchangeSide(message.side),
        timestamp: new Date(message.timestamp).getTime(),
    };
}

---

8. Handle Authentication (If Required)

• Implement authentication mechanisms.
• Sign messages and requests using API keys and secrets.

private signMessage(message: any): any {
    if (this.credential) {
        // Implement signing logic
        // For example, add authentication headers or parameters
    }
    return message;
}

---

9. Implement Reconnection Logic

• Handle WebSocket close events.
• Attempt reconnection after a delay.

this.websocket.on('close', () => {
    setTimeout(() => {
        this.connect(onMessage);
    }, 1000); // Reconnect after 1 second
});

---

10. Implement the stop Method

• Unsubscribe from channels if the exchange supports it.
• Close the WebSocket connection.

public async stop(): Promise<void> {
    const unsubscribeMessage = {
        method: 'UNSUBSCRIBE',
        params: [
            `trades.${this.exchangeSymbol}`,
            `orderbook.${this.exchangeSymbol}`,
            `ticker.${this.exchangeSymbol}`,
        ],
    };
    this.websocket.send(JSON.stringify(unsubscribeMessage));
    this.websocket.close();
}

---

11. Update the Main Public Connector

• Register the new connector in public-connector-main.ts.

// In public-connector-main.ts
const connectorInstance: PublicExchangeConnector = ConnectorFactory.getPublicConnector(
    connectorGroup,
    connectorConfig,
    credential,
);

---

12. Testing and Validation

• Test the connector using the exchange's testnet if available.
• Validate data accuracy by comparing received data with the exchange's official platforms.
• Monitor for errors and edge cases.

---

5. Building a Private Connector

1. Set Up the Connector Class

• Create a new class in src/connectors/private/.
• Implement the PrivateExchangeConnector interface.

import {
    PrivateExchangeConnector,
    ConnectorConfiguration,
    ConnectorGroup,
    Credential,
    Serializable,
} from 'skl-shared';

export class ExchangeNamePrivateConnector implements PrivateExchangeConnector {
    // Implementation
}

---

2. Define Constructor and Member Variables

• Constructor Parameters:

  • group: ConnectorGroup
  • config: ConnectorConfiguration
  • credential: Credential (required for authentication)

• Member Variables:

  • Exchange symbols (exchangeSymbol, sklSymbol)
  • WebSocket and REST endpoints
  • API credentials
  • WebSocket client instance
  • State variables

constructor(
    private group: ConnectorGroup,
    private config: ConnectorConfiguration,
    private credential: Credential,
) {
    this.exchangeSymbol = getExchangeSymbol(this.group, this.config);
    this.sklSymbol = getSklSymbol(this.group, this.config);
}

---

3. Establish the WebSocket and REST Connections

• Initialize WebSocket and REST clients.
• Implement the connect method.

public async connect(onMessage: (messages: Serializable[]) => void): Promise<void> {
    this.websocket = new WebSocket(this.privateWebSocketUrl);

    this.websocket.on('open', async () => {
        await this.authenticate();
    });

    this.websocket.on('message', (data: string) => {
        this.handleMessage(data, onMessage);
    });

    this.websocket.on('error', (error: Error) => {
        // Handle errors
    });

    this.websocket.on('close', () => {
        // Reconnection logic
    });
}

---

4. Handle Authentication

• Implement authentication during WebSocket connection.
• Sign messages and requests using API keys and secrets.

private async authenticate(): Promise<void> {
    // Authentication logic, e.g., sending login messages
    const timestamp = Date.now();
    const signature = this.generateSignature(timestamp);
    const authMessage = {
        op: 'login',
        args: [this.credential.key, timestamp, signature],
    };
    this.websocket.send(JSON.stringify(authMessage));
}

Handling Listen Keys or Session Tokens

• Obtain Listen Key: Before establishing a WebSocket connection, make an authenticated REST API call to obtain a listenKey or session token.
• Include in WebSocket URL: Append the listenKey as a query parameter in the WebSocket URL.
• Manage Listen Keys:
  • Rotation: Implement logic to renew the listenKey before expiration.
  • Cleanup: Delete or invalidate old listen keys to prevent resource leaks.

Example:

private async getListenKey(): Promise<string> {
    const response = await this.postRequest('/userDataStream', {});
    return response.listenKey;
}

public async connect(onMessage: (messages: Serializable[]) => void): Promise<void> {
    const listenKey = await this.getListenKey();
    this.websocket = new WebSocket(`${this.privateWebSocketUrl}?listenKey=${listenKey}`);

    this.websocket.on('open', () => {
        this.startPingInterval();
        this.subscribeToPrivateChannels();
    });

    // Continue with WebSocket setup...
}

---

5. Subscribe to Private Channels

• Subscription Messages: Format subscription messages according to the exchange's private WebSocket API.
• Include Authentication Details: Some exchanges may require additional authentication within the subscription message.

Example:

private subscribeToPrivateChannels(): void {
    const channels = [
        'spot@private.deals.v3.api',
        'spot@private.orders.v3.api',
    ];
    const subscriptionMessage = {
        method: 'SUBSCRIPTION',
        params: channels,
    };
    this.websocket.send(JSON.stringify(subscriptionMessage));
}

---

6. Implement RPC Methods

Implement methods required by the PrivateExchangeConnector interface:

Place Orders

public async placeOrders(request: BatchOrdersRequest): Promise<any> {
    const orders = request.orders.map(order => {
        return {
            symbol: this.exchangeSymbol,
            quantity: order.size.toFixed(8),
            price: order.price.toFixed(8),
            side: mapSide(order.side),
            type: mapOrderType(order.type),
        };
    });

    // Implement order batching if necessary
    // ...

    const endpoint = '/api/v3/order/batch';
    return await this.postRequest(endpoint, { orders });
}

Implementing Order Batching

• Batch Size Limits: Check the exchange's API documentation for any limits on the number of orders per request.
• Batching Logic:
  • Chunk Orders: Split the array of orders into smaller chunks based on the allowed batch size.
  • Sequential Requests: Send batch requests sequentially or in parallel, depending on API rate limits.

Example:

public async placeOrders(request: BatchOrdersRequest): Promise<any> {
    const maxBatchSize = 20; // Example limit
    const orders = request.orders.map(order => ({
        // Map order fields...
    }));

    const batches = this.chunkArray(orders, maxBatchSize);

    const promises = batches.map(batch => {
        return this.postRequest('/batchOrders', { batchOrders: JSON.stringify(batch) });
    });

    const results = await Promise.all(promises);
    return results;
}

private chunkArray(array: any[], chunkSize: number): any[] {
    const results = [];
    for (let i = 0; i < array.length; i += chunkSize) {
        results.push(array.slice(i, i + chunkSize));
    }
    return results;
}

Cancel Orders

public async deleteAllOrders(request: CancelOrdersRequest): Promise<any> {
    const endpoint = '/api/v3/openOrders';
    return await this.deleteRequest(endpoint, { symbol: this.exchangeSymbol });
}

Get Open Orders

public async getCurrentActiveOrders(request: OpenOrdersRequest): Promise<OrderStatusUpdate[]> {
    const endpoint = '/api/v3/openOrders';
    const response = await this.getRequest(endpoint, { symbol: this.exchangeSymbol });

    return response.map(order => ({
        event: 'OrderStatusUpdate',
        connectorType: 'ExchangeName',
        symbol: this.sklSymbol,
        orderId: order.orderId,
        sklOrderId: order.clientOrderId,
        state: mapOrderState(order.status),
        side: mapExchangeSide(order.side),
        price: parseFloat(order.price),
        size: parseFloat(order.origQty),
        notional: parseFloat(order.price) * parseFloat(order.origQty),
        filled_price: parseFloat(order.price),
        filled_size: parseFloat(order.executedQty),
        timestamp: order.time,
    }));
}

Get Balance

public async getBalancePercentage(request: BalanceRequest): Promise<BalanceResponse> {
    const endpoint = '/api/v3/account';
    const response = await this.getRequest(endpoint, {});

    const baseAsset = this.group.name;
    const quoteAsset = this.config.quoteAsset;

    const base = response.balances.find(b => b.asset === baseAsset) || { free: '0', locked: '0' };
    const quote = response.balances.find(b => b.asset === quoteAsset) || { free: '0', locked: '0' };

    const baseBalance = parseFloat(base.free) + parseFloat(base.locked);
    const quoteBalance = parseFloat(quote.free) + parseFloat(quote.locked);

    const baseValue = baseBalance * request.lastPrice;
    const totalValue = baseValue + quoteBalance;
    const inventoryPercentage = (baseValue / totalValue) * 100;

    return {
        event: 'BalanceResponse',
        symbol: this.sklSymbol,
        baseBalance,
        quoteBalance,
        inventory: inventoryPercentage,
        timestamp: Date.now(),
    };
}

---

7. Handle Incoming Messages

• Process messages related to order status updates and account events.
• Use robust parsing to determine event types.

private handleMessage(data: string, onMessage: (messages: Serializable[]) => void): void {
    const message = JSON.parse(data);
    const eventType = this.getEventType(message);

    if (eventType === 'OrderStatusUpdate') {
        const orderStatusUpdate = this.createOrderStatusUpdate(message);
        onMessage([orderStatusUpdate]);
    } else {
        // Handle other events or log unrecognized messages
    }
}

private getEventType(message: any): SklEvent | null {
    if (message.error) {
        logger.error(`Error message received: ${message.error}`);
        return null;
    } else if (message.event === 'subscription') {
        logger.info(`Subscription confirmed: ${JSON.stringify(message)}`);
        return null;
    } else if (message.type === 'orderUpdate') {
        return 'OrderStatusUpdate';
    }
    // Additional event type checks...
    return null;
}

---

8. Implement Reconnection Logic

• Handle WebSocket close events.
• Re-authenticate upon reconnection.
• Ensure subscriptions are re-established.

Ping Mechanism and Cleanup

• Ping Mechanism:

  • Implement a setInterval to send ping messages at the required interval.
  • Keep the WebSocket connection alive as per exchange requirements.

• Cleanup on Reconnection:

  • Clear any existing intervals or timeouts before attempting to reconnect.
  • Reinitialize necessary resources.

Example:

private startPingInterval(): void {
    this.pingInterval = setInterval(() => {
        this.websocket.send(JSON.stringify({ method: 'PING' }));
    }, 10000); // Ping every 10 seconds
}

private stopPingInterval(): void {
    if (this.pingInterval) {
        clearInterval(this.pingInterval);
        this.pingInterval = null;
    }
}

public async connect(onMessage: (messages: Serializable[]) => void): Promise<void> {
    // Existing connection logic...
    this.websocket.on('open', () => {
        this.startPingInterval();
        // Subscribe to channels...
    });

    this.websocket.on('close', () => {
        this.stopPingInterval();
        setTimeout(() => {
            this.connect(onMessage);
        }, 1000); // Reconnect after 1 second
    });
}

---

9. Implement the stop Method

• Unsubscribe from private channels.
• Cancel all open orders if necessary.
• Close WebSocket and clean up resources.

public async stop(): Promise<void> {
    // Unsubscribe from channels
    const unsubscribeMessage = {
        method: 'UNSUBSCRIBE',
        params: [
            `orders.${this.exchangeSymbol}`,
        ],
    };
    this.websocket.send(JSON.stringify(unsubscribeMessage));

    // Optionally cancel all open orders
    await this.deleteAllOrders({
        symbol: this.sklSymbol,
        event: 'CancelOrdersRequest',
        timestamp: Date.now(),
        connectorType: 'ExchangeName',
    });

    // Stop ping interval
    this.stopPingInterval();

    this.websocket.close();
}

---

10. Update the Main Private Connector

• Register the new connector in private-connector-main.ts.

// In private-connector-main.ts
const connectorInstance: PrivateExchangeConnector = ConnectorFactory.getPrivateConnector(
    connectorGroup,
    connectorConfig,
    credential,
);

---

11. Testing and Validation

• Use testnet or sandbox environments if available.
• Use SKL's CLI Locally to test private and public connections
• Verify order placement, cancellation, and updates.
• Ensure balances and open orders are accurately retrieved.
• Monitor for errors and handle edge cases, such as network interruptions or unexpected message formats.

---

6. Best Practices

• Consult Exchange API Documentation: Always refer to the official API docs for the latest information.
• Handle Rate Limits: Be mindful of the exchange's rate limits to avoid throttling or bans.
• Secure Credentials: Never hard-code credentials; use environment variables or secure storage mechanisms.
• Error Handling:
  • Implement comprehensive error handling and logging to facilitate debugging and maintenance.
  • Use try-catch blocks in HTTP requests.
  • Log errors with sufficient detail.
• Code Reusability: Utilize shared utilities and mappings to reduce code duplication and improve maintainability.
• Resource Cleanup: Ensure all resources, such as WebSocket connections and intervals, are properly cleaned up to prevent memory leaks.
• Compliance: Ensure adherence to exchange terms of service and data usage policies.

---

7. Conclusion

By following this guide and leveraging the existing code examples, you should be able to implement both public and private connectors for any exchange. Remember to:

• Understand exchange-specific nuances and adapt your implementation accordingly.
• Test thoroughly in safe environments before deploying to production.
• Ensure compliance with the exchange's terms of service.
• Collaborate with the team: Don't hesitate to reach out if you have questions or need assistance.

Welcome aboard, and happy coding!

---

8. Appendix

Example Code Repositories

• Public Connectors: Refer to existing connectors like Bitmart, MEXC, and Coinbase in our codebase for practical examples.
• Private Connectors: See implementations of Coinbase, Bitmart, and MEXC private connectors for reference.

Utility Functions and Mappings

• Symbol Mapping: getExchangeSymbol, getSklSymbol
• Side Mapping: sideMap, invertedSideMap, MexcSideMap, MexcInvertedSideMap
• Order Type Mapping: orderTypeMap, MexcOrderTypeMap

Common Serializable Types

• Trade
• TopOfBook
• Ticker
• OrderStatusUpdate
• BalanceResponse

---

Note: This guide is intended for internal use within our development team. Please keep all proprietary code and information confidential.

---

End of Guide

Feel free to reach out if you have any questions or need further assistance. No question is a stupid question!