PSync node sequence number increment

[tanim-ics]

In the following code, inside "SetInterval" method node sequence number is increasing every 2s. But I am not understanding it. For example, I want to publish 10 data under 1 ownPrefix which means I want to increase the node sequence number until 10. I do not want to increase the sequence number after every 2s. If I remove the setInterval method then on the consumer side I am not getting the names that are published. Can you please explain a bit?

import {openUplinks} from "@ndn/cli-common";
import {Endpoint} from "@ndn/endpoint";
import {Data, Name} from "@ndn/packet";

import {makePSyncCompatParam, PSyncPartialPublisher, PSyncZlib, SyncNode} from "@ndn/sync";

import * as fs from 'fs';

const syncPrefix = new Name("/sync1");
const ownPrefix = new Name(`/textolines/`);

function text2lines():string[] {
    var text = fs.readFileSync("/Users/XXX/Desktop/NDNts/data.txt", "utf-8");
    return text.split("\n");
}

let lines = text2lines();

//Publish Function
(async () => {

    await openUplinks();
    const sync = new PSyncPartialPublisher({
        p: makePSyncCompatParam({
            ibltCompression: PSyncZlib,
        }),
        // @ts-ignore
        syncPrefix,
    });

    //----------- For debugging output -- optional----------
    if (process.env.NDNTS_SYNC_DEBUG === "1") {
        sync.on("debug", ({ action, interestName }) => {
            console.log(`DEBUG ${action} ${interestName ?? ""}`);
        });
    }

    //---------optional end---------------

    const pub_nodes: Array<SyncNode<Name>> = [];
    for (let i = 0; i < 1; ++i) {
        pub_nodes.push(sync.add(ownPrefix.append(`${i}`)));
    }

    var interval = setInterval(() => {
        for (let i = 0; i<pub_nodes.length; i++)
        {   const node = pub_nodes[i];
            for (let j = 0; j < lines.length; j++)
            {
                node.seqNum++;
                console.log(`PUBLISH ${node.id} ${node.seqNum}`);
            }
        }
    }, 2000);

})().catch(console.error);

//Producer

(async () => {
    await openUplinks();
    const endpoint = new Endpoint();

    endpoint.produce(ownPrefix, async (interest) => {
        console.log(`Got Interest ${interest.name}`);

        const x = Number.parseInt(interest.name.at(2).text, 10);
        console.log(x)
        // Make a Data packet that has the same name as the Interest.
        const data = new Data(interest.name);
        data.freshnessPeriod = 1000;
        data.content = new TextEncoder().encode(lines[x]);
        // Sending the Data is as simple as returning it from the function.
        return data;
    },
// options
    );
})();

another issue is if I change the syncPrefix to syncPrefix_pub then it is throwing me an error which I also do not understand. Why should I use only "syncPrefix" keyword. I am sharing the error:

(node:92568) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'append' of undefined
    at new PSyncPartialPublisher (file:///Users/XXX/Desktop/NDNts/node_modules/@ndn/sync/lib/psync/partial-publisher_node.js:109:54)
    at file:///Users/XXX/Desktop/NDNts/Psync_test.js:16:18
(Use `node --trace-warnings ...` to show where the warning was created)
(node:92568) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:92568) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

[yoursunny]

inside "SetInterval" method node sequence number is increasing every 2s. But I am not understanding it. For example, I want to publish 10 data under 1 ownPrefix which means I want to increase the node sequence number until 10. I do not want to increase the sequence number after every 2s. If I remove the setInterval method then on the consumer side I am not getting the names that are published.

If you have published data for 10 sequence numbers, you should invoke:

node.seqNum += 10;

I have setInterval in several examples to simulate an application periodically publishing data. It isn't the only usage pattern. You can delete setInterval.

---

if I change the syncPrefix to syncPrefix_pub then it is throwing me an error

The PSyncPartialPublisher.Options type requires a syncPrefix property. Not having this property would result in TypeScript compile-time error, or a runtime error if you are writing JavaScript.

You can rename local variable to something else, but the options object must still contain a syncPrefix property, like this:

{
  syncPrefix: syncPrefix_pub,
  // other properties
}

[tanim-ics]

Thank you. I understood and like the way of periodic publishing. I adopted your way. I have now very large number of data to publish (for example: 60K data to publish). But I am now getting another error. Now my setInterval piece of code is like the following:

    const group_names = 50;
    const interval_timeout = 10;

    let interval = setInterval(() => {
        for (let i = 0; i<pub_nodes.length; i++)
        {   const node = pub_nodes[i];
            for (let j = 0; j < group_names; j++)
            {
                if(!(node.seqNum==lines.length)) {
                    node.seqNum++;
                    console.log(`PUBLISH ${node.id} ${node.seqNum}`);
                }
                else {
                    console.log("Publish Complete");
                    clearInterval(interval);
                    break;
                }
            }
        }
    }, interval_timeout);

I get the following error:

failed to asynchronously prepare wasm: RangeError: WebAssembly.instantiate(): Out of memory: wasm memory
RangeError: WebAssembly.instantiate(): Out of memory: wasm memory

but if I change these 2 variables: group_names = 5000 and interval_timeout = 1000 then I am not getting this error. Can you please share your insights about this?

[yoursunny]

failed to asynchronously prepare wasm: RangeError: WebAssembly.instantiate(): Out of memory: wasm memory

PSyncPartial uses a specific Bloom filter that only has C++ implementation. It's compiled as a WebAssembly module. If there are too many calls to the Bloom filter in a short period of time, WebAssembly may run out of memory.

---

If you have lines.length packets to publish, you should do:

node.seqNum = lines.length;

[tanim-ics]

Thank you. I understood now and publishing it periodically with a bit workaround.

I am facing a silly problem. I wrote a class called Producer.ts and I used exportkeyword before the class keyword on that file. I imported into another typescript file called Psync_test.ts. I have compiled with tsc -b.

But when I am running the file node Psync_test.js it is throwing an error that it can not find the module.

XXX@XXX NDNts % node Psync_test.js
internal/process/esm_loader.js:74
    internalBinding('errors').triggerUncaughtException(
                              ^
Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/Users/XXX/Desktop/piccolo/NDNts/Producer' imported from /Users/XXX/Desktop/piccolo/NDNts/Psync_test.js
Did you mean to import ../Producer.js?
    at finalizeResolution (internal/modules/esm/resolve.js:285:11)
    at moduleResolve (internal/modules/esm/resolve.js:708:10)
    at Loader.defaultResolve [as _resolve] (internal/modules/esm/resolve.js:819:11)
    at Loader.resolve (internal/modules/esm/loader.js:89:40)
    at Loader.getModuleJob (internal/modules/esm/loader.js:242:28)
    at ModuleWrap.<anonymous> (internal/modules/esm/module_job.js:73:40)
    at link (internal/modules/esm/module_job.js:72:36) {
  code: 'ERR_MODULE_NOT_FOUND'
}

But the Producer class is on that path. I looked for several solutions but could not solve. Can you please give me any hints?

[yoursunny]

Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/Users/XXX/Desktop/piccolo/NDNts/Producer' imported from /Users/XXX/Desktop/piccolo/NDNts/Psync_test.js Did you mean to import ../Producer.js?

You should always try the advice given in the error message. See also https://nodejs.org/dist/latest-v16.x/docs/api/esm.html#esm_mandatory_file_extensions

[tanim-ics]

failed to asynchronously prepare wasm: RangeError: WebAssembly.instantiate(): Out of memory: wasm memory

PSyncPartial uses a specific Bloom filter that only has C++ implementation. It's compiled as a WebAssembly module. If there are too many calls to the Bloom filter in a short period of time, WebAssembly may run out of memory.

If you have lines.length packets to publish, you should do:

node.seqNum = lines.length;

I am opening this issue again for getting your guidance. I am still not able to find the issue in my implementation. Can you please share your insights how to find out which function is calling the bloom filter too many times and how to debug the wasm memory consumption? Thank you for your patience.

[yoursunny]

If you don't need interop with an existing app, please switch to SvSync, which does not depend on WebAssembly module.

[tanim-ics]

I did not understand what you meant by "If you don't need interop with an existing app". Can you please elaborate?

[yoursunny]

If you don't need interop with an existing app

This means: you are developing a new app, instead of trying to talk to an existing app.