ThatOpen / engine_web-ifc

Reading and writing IFC files with Javascript, at native speeds.
https://thatopen.github.io/engine_web-ifc/demo
Mozilla Public License 2.0
629 stars 198 forks source link

[Bug]: The program exited abnormally while processing the super large model #538

Open verybigzhouhai opened 10 months ago

verybigzhouhai commented 10 months ago

What happened?

Hi, The program exited abnormally while processing the super large model!

Recently, I encountered an unexpected program exit while working on a model containing 300000 components. The error message is shown in the following figure.

Aborted() web-ifc-api-node.js:5175 (node:57980) UnhandledPromiseRejectionWarning: RuntimeError: Aborted(). Build with -sASSERTIONS for more info. at abort (F:\CJXX_WORK\Project\bimrun23dtiles\web-ifc\web-ifc-api-node.js:5179:19) at _abort (F:\CJXX_WORK\Project\bimrun23dtiles\web-ifc\web-ifc-api-node.js:7139:11) at :wasm-function[52]:0x16ff at :wasm-function[345]:0x37baf at :wasm-function[523]:0x76527 at :wasm-function[127]:0xf9b1 at :wasm-function[879]:0xbdbc4 at :wasm-function[889]:0xbe056 at :wasm-function[213]:0x225b9 at :wasm-function[1325]:0xe21f8 (Use node --trace-warnings ... to show where the warning was created) warning.js:43 (node:57980) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1) warning.js:43 (node:57980) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code. image

For this, I created a simple testing program that only writes simple triangular data. When modelCount=150000, the program cannot execute correctly. When modelCount=100000, the ifc file can be successfully output. It should be noted that program abnormal exit does not only occur during saveFile, but also during WriteLine when the modelCount is large enough (300000) test code: `

const fs = require('fs');
const path = require("path");
const { FILE_DESCRIPTION,Schemas,logical,NewIfcModel, ms,IfcAPI, Handle, IFC4, IFC2X3, IFCBUILDINGSTOREY, IFCPROPERTYSINGLEVALUE, IFCSIUNIT, EMPTY, IFCPROPERTYSET, IFCOWNERHISTORY, IFCRELDEFINESBYPROPERTIES } = require("../web-ifc/web-ifc-api-node.js");
const ifcapi = new IfcAPI();

async function createIFC() {

    const filename = "F:\\temp\\ifc\\output\\testexport.ifc"
    await ifcapi.Init();

    if (!fs.existsSync(filename)) {
        if (!fs.existsSync(path.dirname(filename))) {
            fs.mkdirSync(path.dirname(filename), {recursive: true});
        }
    }

    const modelOpton = {
        schema: Schemas.IFC2X3,  // ifc版本
        name: "test.ifc",
        description: ["1", "2"],
        authors: ["3", "4"],
        organizations: ["5", "6"],
        authorization: "78",
    }

    const loaderSettings = {}

    let modelID = ifcapi.CreateModel(modelOpton, loaderSettings);

    const modelCount = 150000;
    for (let i = 0; i < modelCount; i++) {
        console.log(`${i}/${modelCount}`)
        for (let j = 0; j < 50; j++) {
            let cartPoint1 = new IFC2X3.IfcCartesianPoint([1,2,3]);
            let cartPoint2 = new IFC2X3.IfcCartesianPoint([4,5,6]);
            let cartPoint3 = new IFC2X3.IfcCartesianPoint([7,8,9]);

            let array = [cartPoint1, cartPoint2, cartPoint3];

            let poly = new IFC2X3.IfcPolyLoop(array);
            ifcapi.WriteLine(modelID, poly);
        }

    }
    // save file
    fs.writeFileSync(filename, ifcapi.SaveModel(modelID));
    console.log("ifc exported successfully!")
}

function generatorUUID(){
    return Math.random().toString(36).substr(2, 9)+Math.random().toString(36).substr(2, 9)+Math.random().toString(36).substr(2, 4)
}

createIFC()

`

Version

0.0.46

What browsers are you seeing the problem on?

No response

Relevant log output

No response

Anything else?

No response

verybigzhouhai commented 10 months ago

Hi, @beachtom ,Is there any new progress on this issue

beachtom commented 10 months ago

Yes I have identified the cause but still need to code up the fix

verybigzhouhai commented 10 months ago

@beachtom I am glad that the cause of the problem has been identified. Is there an expected timeline? This is important to me. Unfortunately, I am not a C++programmer, otherwise I could have made some contributions. thanks

beachtom commented 10 months ago

I am hoping in the next week.

Sent from Outlook for Androidhttps://aka.ms/AAb9ysg


From: Zhouhai @.> Sent: Friday, January 5, 2024 3:08:58 AM To: IFCjs/web-ifc @.> Cc: Thomas Beach @.>; Mention @.> Subject: Re: [IFCjs/web-ifc] [Bug]: The program exited abnormally while processing the super large model (Issue #538)

External email to Cardiff University - Take care when replying/opening attachments or links. Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor atodiadau neu ddolenni.

@beachtomhttps://github.com/beachtom I am glad that the cause of the problem has been identified. Is there an expected timeline? This is important to me. Unfortunately, I am not a C++programmer, otherwise I could have made some contributions. thanks

— Reply to this email directly, view it on GitHubhttps://github.com/IFCjs/web-ifc/issues/538#issuecomment-1878062442, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGIIJHVAO5CG5DP3JFLFLQLYM5VEVAVCNFSM6AAAAABBCNWVDSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZYGA3DENBUGI. You are receiving this because you were mentioned.Message ID: @.***>

verybigzhouhai commented 10 months ago

Hi, @beachtom ,Sorry to ask again, is there any progress on this issue?

beachtom commented 9 months ago

This is quite a lot more tricky that we thought. Essentially, it seems like you are overfilling the WASM memory so we need to write the data out when that happens.

beachtom commented 9 months ago

So this should now be fixed. I also added a method to write files out using a callback, which makes it a lot quicker in the case of huge files (see the WebIfcApi.spec.ts test lines 536 as an example)

verybigzhouhai commented 9 months ago

@beachtom Thank you very much for making the modifications to this issue. I have tested it and found that the program still exits abnormally when modelCount=30,0000. image

beachtom commented 9 months ago

So this is a different problem - you are now hitting I think a hard limit of 4GB which is what WASM can hold in memory. 150000 is a 2GB IFC file - so 300000 will be a 4GB file so this makes about sense. Realistically is there a use case for a model this big?

verybigzhouhai commented 9 months ago

@beachtom , For complex BIM models, this should be very common. In my two projects, there were over 300000 components, and I could only export 100000. Due to the different number of triangles for each component, in another project, there were 40000 components, but I could only export 8000. Of course, since the model has a large amount of vertex data that can be reused, I have not yet considered this in the program. If there is a 4GB WASM memory limit, how about using this as a node C++ plugin? We usually don't operate such a large IFC file on the web, but in node.js programs, 4GB can easily become a limit

beachtom commented 9 months ago

So the limit only applies for created lines.

So if you are loading a model - you cannot add more than 4GB of IFC lines to it.

Can you describe what exactly you want to do so I can suggest the best way to achieve it - or see how we can fix it.

verybigzhouhai commented 9 months ago

In our system, it is necessary to export existing geometric data to IFC format. These raw data may come from 3D model formats such as. rvt/fbx \ obj, etc. The exported IFC files can be shared with others, so my usage will only be to create IFC files from scratch, not to load existing IFC files. Moreover, most of our model data is complex building information models, so there are many model details, There is a large amount of triangular data

beachtom commented 9 months ago

OK now I understand. I will look into this further.

Do you have any way of predicting the size of the model before you start exporting to IFC?

verybigzhouhai commented 9 months ago

I think the size can be determined by the amount of vertex data, which is known to me before exporting.

verybigzhouhai commented 9 months ago

I may need to use the C++version temporarily, and I noticed that the ifcLoader only has the RemoveLine method and no method to add rows. What if I quickly use the loader to add rows? I have looked at the WriteLine method in web-ifc-api.cpp. Do I need to customize a method for adding rows to ifcLoader? So that I can create, modify, and output IFC files in web-ifc-test.cpp

beachtom commented 9 months ago

So I am working on a node API version. But in the meantime my suggested approach would be to generate two IFC files - start the second one at a higher expressID and then as a post-processing step merge them

verybigzhouhai commented 9 months ago

Okay, I got it!谢谢!

verybigzhouhai commented 9 months ago

I tried to generate a certain amount of data into different files, and when a file is generated, I call ifcap.CloseModel (modelID), and then call ifcapi again Create a new model and obtain a new modelID, but during this process, the memory still dynamically increases, eventually reaching 4GB. Then, the program exits. Did the CloseModel method correctly release the memory? I don't quite understand whether the IfcTokenStream and IfcFileStream sections need to release memory. I added a parameter to control the number of lines in each file, and now it can output 315000 lines, which is not much better than before. image image

    const fs = require('fs');
const path = require("path");
const { FILE_DESCRIPTION,Schemas,logical,NewIfcModel, ms,IfcAPI, Handle, IFC4, IFC2X3, IFCBUILDINGSTOREY, IFCPROPERTYSINGLEVALUE, IFCSIUNIT, EMPTY, IFCPROPERTYSET, IFCOWNERHISTORY, IFCRELDEFINESBYPROPERTIES } = require("../web-ifc/web-ifc-api-node.js");

const ifcapi = new IfcAPI();

async function createIFC() {

    const filename = "F:\\temp\\ifc\\output\\testexport.ifc"
    await ifcapi.Init();

    if (!fs.existsSync(filename)) {
        if (!fs.existsSync(path.dirname(filename))) {
            fs.mkdirSync(path.dirname(filename), {recursive: true});
        }
    }

    const maxModelCountSingleFile = 10000;

    const modelOpton = {
        schema: Schemas.IFC2X3,  // ifc版本
        name: "test.ifc",
        description: ["1", "2"],
        authors: ["3", "4"],
        organizations: ["5", "6"],
        authorization: "78",
    }

    const loaderSettings = {
    } 

    const modelCount = 1000000;
    let modelID = 0;
    let maxExpressID = 1;
    for (let i = 0; i < modelCount; i++) {
        if (i % maxModelCountSingleFile === 0) {
            // save file
            if (i != 0) {
                fs.writeFileSync(filename.replace(".ifc", `${modelID}.ifc`), ifcapi.SaveModel(modelID))
                ifcapi.CloseModel(modelID)
            }

            modelID = ifcapi.CreateModel(modelOpton, loaderSettings);
            console.log(modelID)
        }
        console.log(`${i}/${modelCount}`)
        for (let j = 0; j < 50; j++) {
            let cartPoint1 = new IFC2X3.IfcCartesianPoint([modelID,modelID,modelID]);
            cartPoint1.expressID = maxExpressID++;
            let cartPoint2 = new IFC2X3.IfcCartesianPoint([modelID,modelID,modelID]);
            cartPoint2.expressID = maxExpressID++;
            let cartPoint3 = new IFC2X3.IfcCartesianPoint([modelID,modelID,modelID]);
            cartPoint3.expressID = maxExpressID++;

            let array = [cartPoint1, cartPoint2, cartPoint3];

            let poly = new IFC2X3.IfcPolyLoop(array);
            poly.expressID = maxExpressID++;
            ifcapi.WriteLine(modelID, poly);
        }

    }

    // save file
    fs.writeFileSync(filename.replace(".ifc", `${modelID}.ifc`), ifcapi.SaveModel(modelID)); 
    console.log("ifc exported successfully!")
}

function generatorUUID(){
    return Math.random().toString(36).substr(2, 9)+Math.random().toString(36).substr(2, 9)+Math.random().toString(36).substr(2, 4)
}

createIFC()
beachtom commented 9 months ago

Let me look into this - the memory should be being cleared

beachtom commented 9 months ago

There was a memory leak - I have just fixed it.

If you try what is on main

verybigzhouhai commented 9 months ago

Do I just need to call Ifcap.CloseModel(modelID) to free memory? I downloaded the latest main branch code and compiled it, but the problem still exists. Do you have test cases

beachtom commented 9 months ago

There was another memory leak - I think that is fixed now. Can you let me know

verybigzhouhai commented 9 months ago

The problem still exists, and the total number of rows written has not changed much

verybigzhouhai commented 9 months ago

It should be noted that I do not load the model from a file, but instead create a model from the original. So, does the judgment on line 21 of the IfcTokenChunk.cpp file prevent us from releasing memory https://github.com/IFCjs/web-ifc/blob/8b74ba536a36fce927fa49e75b27291f066191c0/src/cpp/parsing/IfcTokenChunk.cpp#L21

beachtom commented 9 months ago

Well spotted - let me test

verybigzhouhai commented 9 months ago

How to start with a higher expressID? Is the approach here correct?https://github.com/IFCjs/web-ifc/issues/538#issuecomment-1916016068 I seem to have found that incorrect content has been exported in the new file, and I haven't found any patterns yet. It's still being tested.

beachtom commented 9 months ago

So the memory leak is fixed!

To start with a higher ID create an object

let cartPoint1 = ifcApi.CreateIfcEntity(newModID,IFCCARTESIANPOINT,[ifcApi.CreateIfcType(newModID,IFCLENGTHMEASURE,1),ifcApi.CreateIfcType(newModID,IFCLENGTHMEASURE,2),ifcApi.CreateIfcType(newModID,IFCLENGTHMEASURE,3)]);

then override the ID

cartPoint.expressID = XXX;

If this doesn't work let me know the issue and the output and I will fix

verybigzhouhai commented 9 months ago

I have some new findings, the second exported ifc file is not quite correct. As shown in the following figure. image

const fs = require('fs');
const path = require("path");
const { FILE_DESCRIPTION,Schemas,logical,NewIfcModel, ms,IfcAPI,IFCCARTESIANPOINT,IFCFACE,IFCFACEOUTERBOUND,IFCLENGTHMEASURE,IFCPOLYLOOP, Handle, IFC4, IFC2X3, IFCBUILDINGSTOREY, IFCPROPERTYSINGLEVALUE, IFCSIUNIT, EMPTY, IFCPROPERTYSET, IFCOWNERHISTORY, IFCRELDEFINESBYPROPERTIES } = require("../web-ifc/web-ifc-api-node.js");

const ifcapi = new IfcAPI();

async function createIFC() {

    const filename = "F:\\temp\\ifc\\output\\testexport.ifc"
    await ifcapi.Init();

    if (!fs.existsSync(filename)) {
        if (!fs.existsSync(path.dirname(filename))) {
            fs.mkdirSync(path.dirname(filename), {recursive: true});
        }
    }

    const modelOpton = {
        schema: Schemas.IFC2X3,  // ifc版本
        name: "test.ifc",
        description: ["1", "2"],
        authors: ["3", "4"],
        organizations: ["5", "6"],
        authorization: "78",
    }

    const loaderSettings = {
        // LINEWRITER_BUFFER: 100,
        // MEMORY_LIMIT: 4294967295
    } 

    const modelCount = 2;
    let modelID = 0;
    let maxExpressID = 1;
    for (let i = 0; i < modelCount; i++) {

        modelID = ifcapi.CreateModel(modelOpton, loaderSettings);

        console.log(`${modelID}/${modelCount}`)
        const faces = [];

        let cartPoint1 = ifcapi.CreateIfcEntity(modelID,IFCCARTESIANPOINT,[ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,1),ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,2),ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,3)]);
        cartPoint1.expressID = maxExpressID++;
        let cartPoint2 = ifcapi.CreateIfcEntity(modelID,IFCCARTESIANPOINT,[ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,4),ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,5),ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,6)]);
        cartPoint2.expressID = maxExpressID++;
        let cartPoint3 = ifcapi.CreateIfcEntity(modelID,IFCCARTESIANPOINT,[ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,7),ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,8),ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,9)]);
        cartPoint3.expressID = maxExpressID++;
        let array = [cartPoint1, cartPoint2, cartPoint3];
        let poly = ifcapi.CreateIfcEntity(modelID,IFCPOLYLOOP, array);
        poly.expressID = maxExpressID++;
        ifcapi.WriteLine(modelID, poly);

        const faceOuterBound = ifcapi.CreateIfcEntity(modelID,IFCFACEOUTERBOUND, poly, true)
        faceOuterBound.expressID = maxExpressID++;
        ifcapi.WriteLine(modelID, faceOuterBound);

        const face = ifcapi.CreateIfcEntity(modelID,IFCFACE, [faceOuterBound])
        face.expressID = maxExpressID++;
        faces.push(face);
        ifcapi.WriteLine(modelID, face);

        fs.appendFileSync(filename, ifcapi.SaveModel(modelID)); 
        fs.appendFileSync(filename, "\n---------------------\n"); 
        ifcapi.CloseModel(modelID)
    }

    console.log("ifc exported successfully!")

}

createIFC()
beachtom commented 9 months ago

Yes that is a bug you have uncovered. I will take a look hopefully today

verybigzhouhai commented 9 months ago

I have some new findings that multiple exported rows come from duplicate writes, as there is a mutual referencing relationship between IFCCARTESIANPOINT, IfcFaceOuterBound, IfcFace, and IfcClosedShell, so duplicate writes occur in WriteLine. https://github.com/IFCjs/web-ifc/blob/c860b2ca43dce268b581e3248d834c1501e77143/src/ts/web-ifc-api.ts#L598 There are two questions here: The first question, in the example provided above, only those with modelId>0 will have duplicate outputs. The first model will not have this situation in any case, but they all have duplicate writes. The second issue is that when I discover that an object has previous references, we can proactively not execute a writeline on it. However, there is a problem here where many elements are referenced multiple times, such as IfcBuildingStorey and IfcBuildingElementProxy, so it is still impossible to completely avoid them. Alternatively, we can add a property to IfcEntity to determine whether to automatically execute the writeline or cancel the automatic writeline, It is entirely up to the user to decide.

beachtom commented 9 months ago

Yeah your findings match mine. I am not quite sure what is causing it at the moment

beachtom commented 9 months ago

I've fixed the duplicate lines issue!

verybigzhouhai commented 9 months ago

Great!

发自我的iPhone

------------------ Original ------------------ From: Tom Beach @.> Date: Sat,Feb 3,2024 6:27 PM To: IFCjs/web-ifc @.> Cc: Zhouhai @.>, Author @.> Subject: Re: [IFCjs/web-ifc] [Bug]: The program exited abnormally whileprocessing the super large model (Issue #538)

I've fixed the duplicate lines issue!

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

verybigzhouhai commented 9 months ago

Thank you. I'll test it later.

发自我的iPhone

------------------ Original ------------------ From: Tom Beach @.> Date: Sat,Feb 3,2024 6:27 PM To: IFCjs/web-ifc @.> Cc: Zhouhai @.>, Author @.> Subject: Re: [IFCjs/web-ifc] [Bug]: The program exited abnormally whileprocessing the super large model (Issue #538)

I've fixed the duplicate lines issue!

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

verybigzhouhai commented 9 months ago

Great, the program is running well now. I have successfully created a super large IFC file. Thank you very much for your help! At the end of this question, my small suggestion is to add two parameters to the saveFile to determine whether additional information about ifc needs to be output. Because when we export models in batches, in most cases we still need to merge them into one file. If we output each model as a standard ifc file, then we still need to handle a large amount of file merging in the end, which requires time and performance investment. Divide an IFC into three parts. In the first model, only two parts 1 and 2 need to be exported, in the last model, only two parts 2 and 3 need to be exported, and in other models, only the second part needs to be exported. In this way, we can directly write the content returned by the saveFile function to the same file without the need for subsequent processing.

Of course, this is just a suggestion. Thank you again!

image

verybigzhouhai commented 8 months ago

@beachtom Hi, I'm sorry, there are still some issues here. This fix https://github.com/IFCjs/web-ifc/commit/4f8054259c26a1d9cf72163c827bda8be396233c seems to have caused a new memory leak. The memory kept growing slowly until it was not used enough.

beachtom commented 8 months ago

ok let me check that out