Nested arrays, threading and stream support

[formula1]

So, this is probably above my pay grade and probably not useful for most applications but...

• So there is a maximum length of an array
  • while bignumber.js is impressive, it is still limited
• So I wanted to use Petabytes as a reason for this, but a petabyte isn't even that large compared to what bignumber.js can handle
• While I don't know if bignumber.js can handle a monster number, but after a test, yes it can : ) so I guess that can't be used as an excuse
• But regardless, we don't know the future of math and who knows how crazy it may get. While maybe a language like C, C++ or rust is more suited for this kind of thing. Maybe it's better to rewrite the library in wasm. But I think doing things in javascript or typescript is quite cute.

Basic Idea

For adds, multiplies and exponents greater than 1

• the initial number needs to be split up into nested chunks or as the base array
• The other number needs to be split up into nested chunks or as the base array
• We need to be able compare the depth to know how far on each we have have to run the operations on
  • If numberA has a depth or 0 but numberB has a depth of 3
  • the application runs an add operation on numberA.numbers, numberB.numbers[0][0][0]
  • If numberA has a depth of 2 and numberB has a depth of 2
  • the application runs an add operation for each
    • for each depth
    • for each array at that depth
      • if we are at max depth, we run the add operation on those two chunks
• we may need to increase the total depth as well as each nested arrays size

For subtract, divides and exponents less than 1

• the initial number needs to be split up into nested chunks or as the base array if it hasn't been
• The other number needs to be split up into nested chunks or ass the base array if it hasn't been
• We need to be able compare the depth to know how far on each we have have to run the operations on
  • If numberA has a depth or 0 but numberB has a depth of 3
  • the application runs a subtract operation on numberA.numbers, numberB.numbers[0][0][0]
  • If numberA has a depth of 2 and numberB has a depth of 2
  • the application runs a subtract operation for each
    • for each depth
    • for each array at that depth
      • if we are at max depth, we run the subtract operation on those two chunks
• I don't want to get into division and roots and etc
• we may have the opportunity to reduce the total depth as well as each of the nested arrays size

Threading

Since the library is syncronous, it may be a good idea to run it in a background thread for absurd numbers. Maybe store the result number in the main thread but allow a background thread to do all the work. Additionally there may be the opportunity to offload adds to multiple threads, although we do need to handle overflow from a previous chunk to the next chunk making some operations needing to be done twice. What's nice is that threads are available in a variety of javascript environments. WebWorker, Child Process even react native has workers. I know people like immutable instances better than mutable but at least its a start.

Heres some incomplete code relating to some abstraction and convienience functions

// using a symbol because while we don't want anyone outside to be able to use it, we want inter-class communication
const SYM_RUN_OP = Symbol("run operation");

type NumberChunk = Array<number>;
type ChunkHolder = Array<NumberChunk>
type NestedChunks = Array<NestedChunks|ChunkHolder>

type NumbersType = NumberChunk|ChunkHolder|NestedChunk;
type NumberArg = number|BigNumber|AsyncBigNumber;

const SYM_GET_NUM = Symbol("get numbers");
const SYM_RUN_OP = Symbol("run opertation");

class AsyncMutableBigNumber implements BigNumberInterface {
  private #depth: number = 0;
  get depth(){ return this.#depth; };
  private #isPositive: boolean = true;
  get isPositive(){ return this.#isPositive };
  private numbers: NumbersType = [0];
  get [SYM_GET_NUM](){
    return this.numbers;
  }
  private client: BigNumberClient;
  private ready = true;
  constructor(client: BigNumberClient){
    this.client = client;
  }
  private async wrapFn(n: NumberArg, fn: (n: AsyncBigNumber)=>any){
    if(this.ready === false){
      throw new Error("This number is not ready to be calculated. Most operations return a promise you should wait on ");
    }
    this.ready = false;
    const numberB = new AsyncBigNumber(this.client);
    await numberB.set(n)
    await fn(numberB);
    this.ready = true;
    return this;
  }
  setValue(n: NumberArg, b: ValidBase?): Promise<AsyncBigNumber>{
    this.#setValue(prepedNum, b);
    return this;
  }
  private #setValue(n: NumberArg){
    if(this.ready === false){
      throw new Error("This number is in the process of a ")
    }
    // Try to create the initial numbers using a and b
    if(number === void){
      this.numbers = [0];
      return;
    }
    if(typeof n === "number"){
      this.numbers = digestNumber(number);
      this.sign = getSignFromNumber(number)
      return;
    }
    if(n instanceof BigNumber){
      this.numbers = digestBigNumber(number);
      this.sign = getSignFromBigNumber(number)
      return;
    }
    this.numbers = b.numbers;
    this.sign = b.sign;
  }
  plus(n: NumberArg, b: ValidBase?){
    return this.wrapFn(n, (prepedNumber: AsyncBigNumber)=>{
      return this.runOperation("plus", prepedNumber, b);
    })
  }
  minus(n: NumberArg, b: ValidBase?){
    return this.wrapFn(n, (prepedNumber: AsyncBigNumber)=>{
      return this.runOperation("minus", prepedNumber, b);
    })
  }
  private async runOperation(op: string, n: AsyncBigNumber, b: ValidBase?){
    const { numbers, sign } = await this.client[SYM_RUN_OP](op,  this, n, b);
    this.numbers = numbers;
    this.sign = sign;
  }
}

type NumberMessage = {
  op: string,
  a: NumbersType,
  b: NumbersType,
  base: ValidBase,
}
type NumberMessageResult = {
  noError: boolean,
  error: any,
  result: NumbersType
}

type ListenerType = (message: NumberMessage)=>void;

abstract interface WorkerThread {
  abstract listenForMessage(listener: ListenerType): any;
  abstract sendMessage(message: NumberMessage): any;
  abstract terminate(): any;
}

type QueueItem = {
  id: id,
  op: string,
  numberA: AsyncBigNumber,
  numberB: AsyncBigNumber,
  res: (result: NumbersType)=>any,
  rej(e: any)=>any,
};

const SYM_NUMBER_ID = Symbol("number id");

abstract class BigNumberClient {
  private queue: Array<QueueItem> = [];
  private availableWorkers: Array<WorkerThread> = [];
  private keepWorkersAlive: boolean;
  private maxNumWorkers: number;
  private numWorkers = 0;
  private currentWork: {[numId: string]: QueueItem} = {};
  constructor(maxNumWorkers: number, keepWorkersAlive: boolean = true){
    if(maxNumWorkers < 1){
      throw new Error("Need at least one worker when creating a BigNumberClient")
    }
    this.maxNumWorkers = maxNumWorkers;
    if(keepWorkersAlive){
      for(var i = 0; i < maxNumWorkers; i++){
        this.prepWorker();
      }
    }
  }
  protected async abstract createWorker(): WorkerThread;
  private prepWorker(){
     this.numWorkers++;
     this.createWorker().then((worker: WorkerThread)=>{
       worker.listenForMessage((message: NumberMessage)=>{
         this.handleWorkerMessage(worker, message);
       });
      if(this.queue.length > 0){
        return this.makeWorkerDoWork(worker, this.queue.shift());
      }
      this.availableWorkers.push(worker);
    });
  }
  async createBigNumber(a, b){
    const id = createUniqueId();
    const num = new AsyncBigNumber(this);
    await num.set(a, b);
    num[SYM_NUMBER_ID] = id;
    return num
  }
  private handleWorkerMessage(worker: WorkerThread, message: NumberMessage){
    const id = worker[SYM_NUMBER_ID];
    const queueItem = this.currentWork[id];
    if(message.noError) queueItem.res(message.result);
    else queueItem.rej(message.error)
    if(this.queue.length > 0){
      const nextItem = this.queue.shift();
      this.makeWorkerDoWork(worker, nextItem);
      return;
    }
    if(this.keepWorkersAlive) return;
    worker.terminate();
    this.numWorkers--;
  }
  async [SYM_RUN_OP](opType: string, numA: AsyncNumber, numB: AsyncNumber, base: ValidBase){
    return new Promise((res, rej){
      const id = createUniqueId();
      const nextItem = {
        id: id,
        op: opType
        numberA: numA,
        numberB: numB,
        base: base
        res: res, rej: rej
      }
      if(this.availablerWorkers.length > 0){
        return this.makeWorkerDoWork(this.availableWorkers.shift(), nextItem);
      }
      if(this.availableWorkers.length === 0){
        this.queue.push(nextItem);
      }
      if(this.keepWorkersAlive) return;
      if(this.numWorkers < this.maxNumWorkers) this.prepWorker();
    })
  }
  function makeWorkerDoWork(worker: WorkerThread, work: QueueItem){
      worker[SYM_NUMBER_ID] = work.numberA[SYM_NUMBER_ID
      worker.sendMessage({
        op: work.op,
        a: work.numberA[SYM_GET_NUM],
        b: work.numberB[SYM_GET_NUM],
        base: ValidBase,
      })
  }
}

class BigNumberThread {
  abstract listentForNumbers(listener: ListenerType): void;

  abstract sendNumbers(newNumbers: NumberChunk): void;
}

Streams

Nested arrays make things interesting, threading makes things managable but streaming chunks take it to the next level. As a stream, instead of holding giant objects in memory, we can store them in one or multiple files. Instead of sending the entire workload for one worker to do, we can send a peice of work to each of them, get the return value, get the overflow, and send the overflow to the next worker available along with the work result of the next peice. After that, we can pipe the readable stream into a writable stream and that writable writes one or multiple files based off the chunks. Perhaps one NumberChunk per file. perhaps one ChunkHolder per file. Or if the person want to live dangerously, they can turn the stream into the a promise and turn the streaming number into an async big number. Each number that is being worked on could have its own description that gets saved to a folder. When the application loads up, it can read the folder and allow the mathmatician to continue where they left off without having to lose their mind.

I know this is kinda pie in the sky type thinking and I already like what this library is capable of. But I like progress and I think this library has a lot of potential.

A Monster Number in action

If anyones interested in BigNumber handling a monster number, heres some code to get it done. Big number seemed to be able to hande it without issue

import BigNumber from "bignumber.js";

BigNumber.config({
  EXPONENTIAL_AT: 1e+9,
})

var num = new BigNumber(0);

num = num.plus(new BigNumber(2).pow(46));
num = num.times(new BigNumber(3).pow(20))
num = num.times(new BigNumber(5).pow(9));
num = num.times(new BigNumber(7).pow(6));
num = num.times(new BigNumber(11).pow(2));
num = num.times(new BigNumber(13).pow(3));
num = num.times(new BigNumber(17));
num = num.times(new BigNumber(19))
num = num.times(new BigNumber(23))
num = num.times(new BigNumber(29))
num = num.times(new BigNumber(31))
num = num.times(new BigNumber(41))
num = num.times(new BigNumber(47))
num = num.times(new BigNumber(59))
num = num.times(new BigNumber(71))

const recievedValue = num.toString();

console.log("value is:", recievedValue);
const expectedValue = "808,017,424,794,512,875,886,459,904,961,710,757,005,754,368,000,000,000".replace(/,/g, "");
console.log("expected value:", expectedValue);

console.log("are they equal?", recievedValue === expectedValue); // true

[formula1]

Perhaps this is too ambitious at the moment. Would love some feedback but this isn't a priority and nobody probably is in dire need of any of these features.

I'll let the issues be real issues and let this sit. It can still be found if someone is interested