With @web-ifc we can split the Uint8Array into chunks to read?

[RyugaRyuzaki]

What is your idea?

• The initial idea is that I have 1 large ifc (4GB), I can split it into many Uint8Arrays (I temporarily set them as a1, a2, a3...) Now I want to read it into parts as follows: api.OpenModel(a1), api.OpenModel(a2).... And is this solution applicable to the current web-ifc version?

[beachtom]

So you can pass in a callback function into web-ifc:

OpenModelFromCallback(callback: ModelLoadCallback , settings?: LoaderSettings): number

Where ModelLoadCallback is (offset:number, size: number) => Uint8Array - you can provide a function that will hookup that function to get the data from the right UInt8Array

[RyugaRyuzaki]

So you can pass in a callback function into web-ifc:

OpenModelFromCallback(callback: ModelLoadCallback , settings?: LoaderSettings): number

Where ModelLoadCallback is (offset:number, size: number) => Uint8Array - you can provide a function that will hookup that function to get the data from the right UInt8Array

Here i my code :

import * as WEBIFC from "web-ifc";
import * as fs from "fs";
import {IfcFragmentSettings} from "./parser/IfcFragmentSettings";

const filePath = "./input/large192.ifc";
const highWaterMark = 200 * 1024 * 1024;

class App {
  private settings = new IfcFragmentSettings();
  private _webIfc = new WEBIFC.IfcAPI();
  async streamFromBufferCallBack(callback: WEBIFC.ModelLoadCallback) {
    const before = performance.now();
    await this.readIfcFile(callback);

    this.cleanUp();

    console.log(`Streaming the IFC took ${performance.now() - before} ms!`);
  }
  private async readIfcFile(callback: WEBIFC.ModelLoadCallback) {
    const {path, absolute, logLevel} = this.settings.wasm;
    this._webIfc.SetWasmPath(path, absolute);
    await this._webIfc.Init();
    if (logLevel) {
      this._webIfc.SetLogLevel(logLevel);
    }
    this._webIfc.OpenModelFromCallback(callback, this.settings.webIfc);
  }
  private cleanUp() {
    (this._webIfc as any) = null;
    this._webIfc = new WEBIFC.IfcAPI();
  }
}

const app = new App();

(async () => {
  const exist = fs.existsSync(filePath);
  if (!exist) {
    console.error("File not found");
    process.exit(1);
  }
  try {
    let offset = 0;
    const readStream = fs.createReadStream(filePath, {
      highWaterMark,
    });

    readStream.on("data", (chunk: Buffer) => {
      const modelCallBack: WEBIFC.ModelLoadCallback = (
        callbackOffset: number,
        size: number
      ) => {
        // how can i calculate here?
        return new Uint8Array(chunk); //
      };
      offset += chunk.byteLength;
    });

    readStream.on("end", async () => {});

    readStream.on("error", (error: any) => {
      console.log(error);
      process.exit(1);
    });
  } catch (error) {
    console.log(error);
    process.exit(1);
  }
})();

Any idea for modelCallBack: WEBIFC.ModelLoadCallback ? thank @beachtom

[beachtom]

Are you running in node or in the browser?

[RyugaRyuzaki]

Are you running in node or in the browser?

hi, thank for quick reply. i'm running on Nodejs

[beachtom]

This one reads from an open file - streaming into web-ifc

let retriever = function (offset:number, size: number) { let data = new Uint8Array(size); let bytesRead = fs.readSync(file,data,0,size,offset);
if (bytesRead <= 0 ) return new Uint8Array(0); return data;
}

[RyugaRyuzaki]

This one reads from an open file - streaming into web-ifc

let retriever = function (offset:number, size: number) { let data = new Uint8Array(size); let bytesRead = fs.readSync(file,data,0,size,offset); if (bytesRead <= 0 ) return new Uint8Array(0); return data; }

@beachtom great. i did it.Thank! . However is there any way to control offset and size ? Because i test with model 3gb each callback just 67mb..

[RyugaRyuzaki]

@beachtom thank you, can you tell me how can i integrate in browser?. Here my code and it thrown error .

 async function stream(payLoad: File) {
  const reader = payLoad.stream().getReader({mode: 'byob'});
  let offset = 0;

  const modelCallback = async (_callbackOffset: number, size: number) => {
    if (size > payLoad.size) size = payLoad.size;
    const data = new Uint8Array(size);
    while (offset < size) {
      const remainingSize = payLoad.size - offset;
      const view = new Uint8Array(remainingSize);
      const {value, done} = await reader.read(view);
      if (done) break;
      if (value && value.byteLength > 0) {
        data.set(value, offset);
        offset += value.byteLength;
      }
    }
    console.log(data);
    return data;
  };
  //@ts-ignore
  await new PropertyLoader(onPropertiesLoaded).streamFromCallBack(modelCallback);
}

error :

RuntimeError: memory access out of bounds
    at web-ifc.wasm:0xcf6a2
    at web-ifc.wasm:0xc6100
    at Object.OpenModel (eval at newFunc (chunk-XC6DDJAD.js?v=63910048:7308:31), <anonymous>:9:10)
    at IfcAPI2.OpenModelFromCallback (chunk-XC6DDJAD.js?v=63910048:66577:34)
    at PropertyLoader.streamIfcFile (PropertyLoader.ts:79:17)
    at async PropertyLoader.streamFromCallBack (PropertyLoader.ts:54:5)
    at async stream (IfcPropertyWorker.ts:35:3)
    at async onLoad (IfcPropertyWorker.ts:43:5)

[beachtom]

To increase the callback site - change the TAPE_SIZE parameter inside Loader Settings

Is the model opening in node. Opening large models in the browser is very difficult as the browsers tend to run out of memory - it is often better to convert to fragments and then use from there

[RyugaRyuzaki]

To increase the callback site - change the TAPE_SIZE parameter inside Loader Settings

Is the model opening in node. Opening large models in the browser is very difficult as the browsers tend to run out of memory - it is often better to convert to fragments and then use from there

Yes, in node, i set the function you gave me and it worked as well. but in browser it throws error

[beachtom]

I suspect the file may well be just too big for the browser to load

[RyugaRyuzaki]

I suspect the file may well be just too big for the browser to load

@beachtom all file I tested in browser same throw error. In node, all are ok even which ~4gb.

[beachtom]

Can you test if the actual callback works in the browser? By adding some test code i.e. calling

await modelCallback(0,66000) or something

[RyugaRyuzaki]

Can you test if the actual callback works in the browser? By adding some test code i.e. calling

await modelCallback(0,66000) or something

@beachtom Sure, here is my code:

export async function stream(payLoad: File) {
  const reader = payLoad.stream().getReader({mode: 'byob'});
  let offset = 0;

  const modelCallback = async (_callbackOffset: number, size: number) => {
    if (size > payLoad.size) size = payLoad.size;
    const newSize = size > payLoad.size ? payLoad.size : size;
    const data = new Uint8Array(newSize);
    console.log('File Size', payLoad.size);
    while (offset < newSize) {
      const remainingSize = payLoad.size - offset;
      const view = new Uint8Array(remainingSize);
      const {value, done} = await reader.read(view);
      if (done) break;
      if (value && value.byteLength > 0) {
        data.set(value, offset);
        offset += value.byteLength;
      }
    }
    return data;
  };
  const bytes = await modelCallback(0, 66 * 1024 * 1024);
  console.log('All Bytes Read', bytes);
  //@ts-ignore
  // await new PropertyLoader(onPropertiesLoaded).streamFromCallBack(modelCallback);
}

And here is result :

Another file : (45MB) And if i change size =20MB

[beachtom]

are you able to share your code with me to take a look? Or I can provide you with debug wasm files that should help us debug exactly where in the WASM the error is occuring

[RyugaRyuzaki]

are you able to share your code with me to take a look? Or I can provide you with debug wasm files that should help us debug exactly where in the WASM the error is occuring

Hi @beachtom here is the code https://drive.google.com/file/d/1n_zYBnWdZB1LRAnzUQrxsvud0bHR5-8a/view?usp=sharing