API - add timeout to fileWrite wrapper

[jaros1]

See issues https://github.com/jaros1/Money-Network-W3/issues/12 and https://github.com/HelloZeroNet/ZeroNet/issues/1391

Wait max 1 second for fileWrite cmd to complete. Timeout. check:

• has file been written to file system? Use fileList cmd
• is file in bad files? Use mergerSiteList cmd

[jaros1]

Added timeout in fileWrite wrapper z_file_write:

        // fileWrite timeout in 1 second (UI error: This file still in sync, if you write it now, then the previous content may be lost)
        // check fileList and mergerSiteList after fileWrite timeout
        cb2_timeout = function () {
            var pgm = module + '.z_file_get cb2_timeout: ' ;
            var directory ;
            if (cb2_done) return ; // cb2 has already run
            // timeout.
            console.log(pgm + 'issue #359 API - add timeout to fileWrite wrapper') ;
            console.log(pgm + 'fileWrite timeout after 1 second') ;

            // 1: check if file has been written to file system
            directory = get_merged_type() + '/' + hub + '/data/users/' + auth_address ;
            ZeroFrame.cmd("fileList", directory, function (files) {
                var pgm = module + '.z_file_get cb2_timeout fileList callback 1: ' ;
                console.log(pgm + 'Looking for ' + inner_path + ', files = ' + JSON.stringify(files)) ;

                // 2: check if file is in bad_files.
                ZeroFrame.cmd("mergerSiteList", [true], function (merger_sites) {
                    var pgm = module + '.z_file_get cb2_timeout mergerSiteList callback 2: ' ;
                    if (merger_sites.error) {
                        console.log(pgm + 'mergerSiteList failed with ' + merger_sites.error) ;
                        return cb2(null, true) ;
                    }
                    if (!merger_sites[hub]) {
                        console.log(pgm + 'error. could not find hub ' + hub + ' in merger_sites ' + JSON.stringify(merger_sites)) ;
                        return cb2(null, true) ;
                    }

                    console.log(pgm + 'Looking for ' + inner_path + ' in bad files. site_info = ' + JSON.stringify(merger_sites[hub])) ;
                    if (cb2_done) return ; // cb2 has already run
                    cb2(null, true) ;

                }) ; // mergerSiteList callback 2

            }) ; // fileList callback 1

        }; // cb2_timeout
        process_id = setTimeout(cb2_timeout, 1000) ;

[jaros1]

No joy. fileList cmd is also hanging :-(

[jaros1]

Tried to fix problem with manual sign remove_missing_optional + publish: siteSign 1HXzvtSLuvxZfh6LgdaqTk4FSVf7x8w7NJ --inner_path data/users/18DbeZgtVCcLghmtzvg4Uv8uRQAwR8wnDQ/content.json --remove_missing_optional --publish Fixed the problem, but is not a solution!

[jaros1]

Second workaround. Try above workaround once. Second hanging fileWrite. Notofication. Ask user to "cleanup" bad files ... :-/

[jaros1]

Changed fileWrite timeout wrapper. Trying to fix hanging fileWrite with publish ...

        // fileWrite timeout in 1 second (UI error: This file still in sync, if you write it now, then the previous content may be lost)
        // check fileList and mergerSiteList after fileWrite timeout
        cb2_timeout = function () {
            var pgm = module + '.z_file_get cb2_timeout 1: ' ;
            var directory, count, inner_path1 ;
            if (cb2_done) return ; // cb2 has already run
            // timeout.
            console.log(pgm + 'issue #359 API - add timeout to fileWrite wrapper') ;
            console.log(pgm + 'fileWrite timeout after 1 second.') ;

            // workaround 1: try sign with remove_missing_optional + publish
            // workaround 2: notification only
            directory = 'merged-' + get_merged_type() + '/' + hub + '/data/users/' + auth_address ;
            count = z_file_write_hanging[directory] || 0 ;
            count++ ;
            z_file_write_hanging[directory] = count ;
            if (count == 1) {
                // maybe content.json is in list of bad files. Try if publish solved the problem
                console.log(pgm + 'trying if sign with remove_missing_optional + publish will solve the problem') ;
                inner_path1 = directory + '/content.json' ;
                z_site_publish({inner_path: inner_path1, remove_missing_optional: true, reason: 'hanging fileWrite', encrypt: options.encrypt}, function (res) {
                    if (res == 'ok') ZeroFrame.cmd("wrapperNotification", ['info', 'Problem with hanging fileWrite operation may have been solved', 5000]);
                    else console.log(pgm + 'publish failed. error = ' + JSON.stringify(res)) ;
                }) ;
            }
            else {
                console.log(pgm + 'hanging fileWrite count = ' + count) ;
                ZeroFrame.cmd("wrapperNotification", ['info', 'Warning. Hanging fileWrite operation
Check bad_files for hub ' + hub + '
for auth_address ' + auth_address]);
            }

        }; // cb2_timeout 1
        process_id = setTimeout(cb2_timeout, 1000) ;

[jaros1]

Still problem when starting comp B after comp A in the morning. Workaround is not working. Forgot to start browser console and forgot to keep browser log before MN page reload. Publish in MN is running forever.

content.json and data.json is in bad_files list: issue-359-sites-json.log

MN warning:

w3-127.0.0.1-1524470162369.log w2-127.0.0.1-1524470141095.log mn-127.0.0.1-1524470128183.log

[jaros1]

Added more text to log and better notifications. Workarounds is: 1) publish in MoneyNetworkAPI 2) manual sign and publish. cmd in log and notification 3) clear bad_files in sites.json cache

[jaros1]

Notification in W2 (torando).

But no "This file still in sync, if you write it now, then the previous content may be lost" notification! But many open/close web socket connection. Mayby an other issue!

127.0.0.1-1524581384703.log

[jaros1]

fileWrite timeout notification in W2 (torando) :

• timeout in MoneyNetworkAPI after 1 second. fileWrite operation completed without error after 1827 ms
• fileWrite should return "timeout" and not "ok" after timeout

w2-127.0.0.1-1524642897413.log

[jaros1]

Problem with sync file. Workaround in MoneyNetworkAPI did not help. Found 2 bad files after JS workaround: "data/users/18DbeZgtVCcLghmtzvg4Uv8uRQAwR8wnDQ/content.json": 5, "data/users/18DbeZgtVCcLghmtzvg4Uv8uRQAwR8wnDQ/e46aad1bc3-i.1524647946060": 1,

Did terminal workaround: ./ZeroNet.sh siteSign 1HXzvtSLuvxZfh6LgdaqTk4FSVf7x8w7NJ --inner_path data/users/18DbeZgtVCcLghmtzvg4Uv8uRQAwR8wnDQ/content.json --remove_missing_optional --publish bad-files was not removed.

starting ui-server. bad-files was not removed.

reloaded w2 page. bad_files was not removed but no hanging fileWrite operation and everything is OK ...

127.0.0.1-1524648508978.log

[jaros1]

Changed timeout in MoneyNetworkAPI to 5 seconds. z_file_write will return timeout and not ok after hanging fileWrite operations

[jaros1]

Old laptop (spain). MN notification:

127.0.0.1-1524716774829.log

[jaros1]

New laptop "torando". W2 file sync error but no hanging fileWrite notification. Check also for hanging dbQuery in log!

127.0.0.1-1524991123927.log

todo:

• hanging dbQuery. OK response after 63 seconds. Must be connected hanging fileWrite operation?
• sitePublish. OK in UI server log but callback in JS was never executed. Hanging fileWrite notification in sitePublish callback and therefore no hanging fileWrite notification in UI. Move notification to before siteSign and sitePublish.

Refactored some code in MoneyNetworkAPI (z_file_get and z_file_write). Now notification before trying to fix hanging fileWrite with JS sign and publish

[jaros1]

Now hanging fileWrite in W3, but nice notifications before and after trying to fix the problem with publish

[jaros1]

Nothing more to do here. Problem must be fixed in ZeroNet ...