How to convert uint8array data to BTreeMap or vector/array using typescript.

[SachinNandal]

Hello, I'm serializing a BTreeMap using BorshSerialize in rust. How to deserialize this uint8array serialized data to vector/array or BTreeMap in Typescript using borsh-js or any other library? Please help

Below is my rust code

#[derive( Clone, BorshDeserialize, BorshSerialize, Copy)]
pub struct BuyOrderData {
    pub owner_public_key: [u8;32],
    pub receiving_bc_public_key: [u8;32],
    pub amount_qc: u64,
}

#[derive(Default)]
pub struct BuyOrderDataBTreeMap {
    is_initialized: bool,
    bids:  BTreeMap<u128, BuyOrderData>,
}

const INITIALIZED_BYTES: usize = 1;
const BTREEMAP_LENGTH: usize = 4;
const BTREEMAP_STORAGE: usize = 8600;
const ORDER_DATA_HEAP_ACCOUNT_STATE_SPACE: usize = INITIALIZED_BYTES + BTREEMAP_LENGTH + BTREEMAP_STORAGE;

impl BuyOrderDataBTreeMap {
    pub fn set_initialized(&mut self) {
        self.is_initialized = true;
    }

    pub fn add(&mut self, new_key:u128, new_owner_public_key:  [u8;32], new_receiving_bc_public_key:  [u8;32],  new_amount_qc: u64) {
        self.bids.insert(new_key, BuyOrderData {owner_public_key: new_owner_public_key, receiving_bc_public_key: new_receiving_bc_public_key, amount_qc: new_amount_qc});
    }

    pub fn remove(&mut self, key: u128) {
        self.bids.remove(key);
    }

}

impl Sealed for BuyOrderDataBTreeMap {}

impl IsInitialized for BuyOrderDataBTreeMap {
    fn is_initialized(&self) -> bool {
        self.is_initialized
    }
}

#[allow(clippy::ptr_offset_with_cast)]
impl Pack for BuyOrderDataBTreeMap {
    const LEN: usize = ORDER_DATA_HEAP_ACCOUNT_STATE_SPACE;
    fn unpack_from_slice(src: &[u8]) -> Result<Self, ProgramError> {
        let src = array_ref![src, 0, BuyOrderDataBTreeMap::LEN];
        let (is_initialized_src, data_len_src, data_src) = array_refs![src, INITIALIZED_BYTES, BTREEMAP_LENGTH, BTREEMAP_STORAGE];

        let is_initialized = match is_initialized_src {
            [0] => false,
            [1] => true,
            _ => return Err(ProgramError::InvalidAccountData),
        };
        let data_len = u32::from_le_bytes(*data_len_src) as usize;
        if data_len == 0 {
            Ok(BuyOrderDataBTreeMap {
                is_initialized,
                bids: BTreeMap::<u128, BuyOrderData>::new(),
            })
        }  else {
            let data_dser =
                BTreeMap::<u128, BuyOrderData>::try_from_slice(&data_src[0..data_len]).unwrap();
            Ok(BuyOrderDataBTreeMap {
                is_initialized,
                bids: data_dser,
            })
        }
    }

    fn pack_into_slice(&self, dst: &mut [u8]) {
        let dst = array_mut_ref![dst, 0, BuyOrderDataBTreeMap::LEN];
        let (is_initialized_dst, data_len_dst, data_dst) = mut_array_refs![dst, INITIALIZED_BYTES, BTREEMAP_LENGTH, BTREEMAP_STORAGE];
        is_initialized_dst[0] = self.is_initialized as u8;
        let orders_store_heap_data = self.bids.try_to_vec().unwrap();
        let data_len = orders_store_heap_data.len();
        data_len_dst[..].copy_from_slice(&(data_len as u32).to_le_bytes());
        data_dst[0..data_len].copy_from_slice(&orders_store_heap_data);

    }
}

[willemneal]

There is a lot to unpack here.

Firstly, it seems that contract has no public methods, that is it should be:

#[near_bindgen]
struct BuyOrderDataBTreeMap {
...
}

#[near_bindgen]
impl BuyOrderDataBTreeMap {
// This methods are public with `pub`

The external methods encode the data returned as JSON by default; you can also choose to encode them as borsh (see here), but it is harder to decode as you are trying to do here.

Secondly, if your goal is to have some public methods that update the state and then pull the state via RPC to then decode, or as suggested above return the value encoded as borsh, you will need a schema in JS that matches the schema of the data in the contract. This isn't easy as you must hand write the schema see here.

The other aspect of your code example is that it seems you are trying to store the entire BTree as one entry in storage. This will mean having to read all of it back each time and write it back each time, which as the tree grows will get more and more expensive. So instead I suggest using the provided TreeMap , which stores each node of the tree separately in storage meaning that each time you make a write it's O(log(n)) of the size and reads are O(1).

[SachinNandal]

Thank You So Much for reply @willemneal

Above code that I added here is a part of my solana program. As solana programs are stateless therefore I am using borsh to serialize and deserialize my data so that I can store it on solana blockchain. And whenever I make a call using web3js to my program/smart contract to add/remove anything from my BTreeMap stored on solana blockchain, I have to fetch full BTreeMap. If I use TreeMap instead of BTreeMap, is it possible to add/remove data from TreeMap without loading entire TreeMap data into my solana program?

And after using this TreeMap in rust, is there any typescript schema of TreeMap to decode my uint8array data(which I read from blockchain) to TreeMap<key, BuyOrderDataTreeMap>?

[willemneal]

Sorry I'm really not familiar with Solana. There is no schema of TreeMap as its general use case is for view contract calls to access only parts that are needed. TreeMap is designed for Near anyway as it reads and writes from the storage API.

I'm still not sure what your Solana's external contract calls are. Your best bet is to see the example link and make your own schema or use another binary encoding that is more common in the Solana ecosystem.

[SachinNandal]

As it's very difficult and time consuming to write Typescript schema of rust BTreeMap. I decided to build my rest api to read blockchain data using Rust instead of Typescript.

Thanks for your time @willemneal