File system not updated

[mcadam]

Hello,

I am still testing the drive, speeds are great, uploads are going well, no problem there but I am seeing a weird behaviour on updates I think.

To give you more context, I am running in a cluster there is one node that writes on the disk, upload some files. On the other node I should see the files appearing after 5 minutes waiting on the next update.

Whats happening it looked it work sometimes but not all the time.

On the node uploading the file, I can go to the folder and see the new file using ls command no problem there. On the new node though, the file is not present, I checked the logs in mongodb and I can see an update command on the file missing:

2018-09-08T06:35:27.656+0000 I WRITE    [conn3] update DriveFS.GDriveData command: { q: { id: "1rhjqb64HMr4lUwjWlz3sESzXXzb46TYC" }, u: { $set: { kind: "drive#file", id: "1rhjqb64HMr4lUwjWlz3sESzXXzb46TYC", ...

If I restart the drive on that node then all the files appear on the file system.

Any ideas?

I am running the drive with those options:

 -f --fuse-allow-other --cache-disk-size=10240 --cache-size=2048 --max-concurrent-downloads=24

Thanks for your help, even though I have this issue again its very promising so far :)

[EricTheMagician]

i pushed some new code. Can you give it a try. It's not been mission critical for me, but I think I found a hole in the logic that should fix it.

I also parameterized the refresh interval, so you can set it to lower to test it more quickly.

[mcadam]

I am having an error on compiling

Scanning dependencies of target DriveFS
[ 58%] Building CXX object CMakeFiles/DriveFS.dir/src/main.cpp.o
[ 62%] Building CXX object CMakeFiles/DriveFS.dir/src/gdrive/File.cpp.o
[ 65%] Building CXX object CMakeFiles/DriveFS.dir/src/gdrive/Filesystem.cpp.o
[ 68%] Building CXX object CMakeFiles/DriveFS.dir/src/BaseAccount.cpp.o
[ 72%] Building CXX object CMakeFiles/DriveFS.dir/src/gdrive/Account.cpp.o
[ 75%] Building CXX object CMakeFiles/DriveFS.dir/src/gdrive/FileIO.cpp.o
/app/src/main.cpp: In function 'int main(int, char**)':
/app/src/main.cpp:257:45: error: 'DriveFS::Filesystem' has not been declared
     struct fuse_lowlevel_ops ops = DriveFS::Filesystem::getOps({});
                                             ^~~~~~~~~~
make[2]: *** [CMakeFiles/DriveFS.dir/build.make:76: CMakeFiles/DriveFS.dir/src/main.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:106: CMakeFiles/DriveFS.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

[mcadam]

Changed it back to

-    struct fuse_lowlevel_ops ops = DriveFS::Filesystem::getOps({});
+    struct fuse_lowlevel_ops ops = DriveFS::getOps();

and could compile again, is this ok?

[mcadam]

Apparently not ?

2018-09-09 09:18:33,422 [T] [Account.cpp:105] Getting Account
2018-09-09 09:18:33,424 [I] [Account.cpp:118] Access tokens founds
2018-09-09 09:18:33,429 [I] [Account.cpp:51] Previous change tokens founds
2018-09-09 09:18:33,429 [I] [BaseAccount.cpp:137] Refreshing access tokens
2018-09-09 09:18:33,578 [I] [Account.cpp:579] Getting updated list of files and folders
2018-09-09 09:18:33,579 [D] [Account.cpp:580] teamDriveId root and token
2018-09-09 09:18:34,253 [T] [Account.cpp:434] Filling GDrive file information from cache
2018-09-09 09:18:36,323 [T] [Account.cpp:568] idToObject has 38262 items.
2018-09-09 09:18:36,323 [T] [Account.cpp:570] inodeToObject has 38262 items.
2018-09-09 09:18:36,323 [T] [Account.cpp:406] Linking parent and children
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_any_cast> >'
  what():  boost::bad_any_cast: failed conversion using boost::any_cast
2018-09-09 09:18:36,786 [F] [easylogging++.cc:2868] CRASH HANDLED; Application has crashed due to [SIGSEGV] signal
    ======= Backtrace: =========
    [1] /app/DriveFS(+0x744e4) [0x559dfebf24e4]
    [2] /app/DriveFS(+0x746c5) [0x559dfebf26c5]
    [3] /usr/lib/libc.so.6(+0x37e00) [0x7f9cba4fee00]
    [4] /usr/lib/libc.so.6(abort+0x201) [0x7f9cba4e974e]
    [5] /usr/lib/libstdc++.so.6(+0x8958e) [0x7f9cba8b558e]
    [6] /usr/lib/libstdc++.so.6(+0x8fdfa) [0x7f9cba8bbdfa]
    [7] /usr/lib/libstdc++.so.6(+0x8fe57) [0x7f9cba8bbe57]
    [8] /usr/lib/libstdc++.so.6(+0x900ac) [0x7f9cba8bc0ac]
    [9] /app/DriveFS(+0xaf192) [0x559dfec2d192]
    [10] /app/DriveFS(+0xad298) [0x559dfec2b298]
    [11] /app/DriveFS(+0xaaeee) [0x559dfec28eee]
    [12] /app/DriveFS(+0xa8fee) [0x559dfec26fee]
    [13] /app/DriveFS(+0xa0241) [0x559dfec1e241]
    [14]  0xf3) [0x7f9cba4eb223]:__libc_start_main+0xf3) [0x7f9cba4eb223]
    [15] /app/DriveFS(+0x63cce) [0x559dfebe1cce]

2018-09-09 09:18:36,786 [W] [easylogging++.cc:2868] Aborting application. Reason: Fatal log at [/app/external/easyloggingpp/src/easylogging++.cc:2868]

[mcadam]

My bad I renamed the new option refresh_interval to refresh-interval to be consistent with the format of the other options and forgot to change

-    account.setRefreshInterval(vm["refresh_interval"].as<int>());
+    account.setRefreshInterval(vm["refresh-interval"].as<int>());

Now its running again, will keep you posted if I still any update issues.

Thanks

[mcadam]

Hello,

Don't forget to update that also to rename the option

-    account.setRefreshInterval(vm["refresh_interval"].as<int>());
+    account.setRefreshInterval(vm["refresh-interval"].as<int>());

So I've had the new code running for a day now, and I am again missing some new files not appearing on the other drive. It was working fine for a while but now I have to restart the drive to see the new files appearing on the file system.

[EricTheMagician]

Good catch on the refrfesh_initerval.

Also, it might not have updated the new folder if a file had moved. Would that make sense?

Anyways, I'll have to keep looking later and continue refactoring.

[mcadam]

What do you mean by if a file had moved? What is happening is one drive is creating a new file, uploading it. On the other drive I receive updates from the changes list, sometimes the files is then not appearing in the file system and sometimes its fine.

[EricTheMagician]

By move, I meant move a file from one folder to another, which is not the case here.

There's one more thing I can try later.

How often were you having the problem? Or is it sporadic and random now?

[mcadam]

Pretty hard to say, I thought maybe it was happening when I filled up the cache and maybe it was messing with things, but does not look like it, just after a while there are some files not appearing, let me try to add new ones and see if they appear now

[EricTheMagician]

For the missing files, do you see it in the mongo log?

[mcadam]

I am not sure, I can see some new files appearing in the disk but dont find any logs with their names in Mongo, not sure Mongo logs all the time when creating new file the name, only on updates?

So after running it for a few days, I can see some new files appearing fine and some others not. I can see on one side, "Successfully uploaded" but not appearing on the other disk while sometimes working fine.

For now I need to restart the disk and then the missing ones appears.

[mcadam]

Hi,

I think the issue is when there are multiple updates in the same changes query on the drive, if i get one update at a time per changes query I think the files are appearing ok. Not 100% sure but maybe that helps for your investigation?

[EricTheMagician]

Hi mcadam, I added an extra function call to invalidate the entry. If that doesn't work, I'll try uploading 1k files to see how many are missed.

[mcadam]

Thanks will try to compile the new version and deploy that to my cluster and give you some feedback in the next few days.

Thanks again for your work, already a great tool :)

[mcadam]

Hello,

After using the new version for a while, it's still not getting every updates, still need to restart the drives to get updates on some of the files. I think that version I might even get less updates, but couldn't say for sure.

[mcadam]

I think when there is a creation of a new directory involved it doesn't work, it seems that new files added to existing directories are appearing fine but if there is a new directory in which the new file is going then it doesn't work? Not sure about that, just a thought from what I can see.

[mcadam]

Yeah I don't know, I just created a file on one drive, then waited for the next update on a different drive, I can see the token increasing so there was some update events in there but the file is not appearing, looks pretty random.

[EricTheMagician]

It should be fixed now, but please give it a try

[mcadam]

Cool thanks will let you know if the next couple days how it goes, thanks again for your work, really appreciated 👍

[mcadam]

Hello, just to let you know it seems to be going very well on updates now, will let you know again in a week if we can close this.

[EricTheMagician]

Great! I'm glad to hear.

On Wed, Oct 24, 2018, 8:39 AM Adam Guldemann notifications@github.com wrote:

Hello, just to let you know it seems to be going very well on updates now, will let you know again in a week if we can close this.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/thejinx0r/DriveFS/issues/6#issuecomment-432638354, or mute the thread https://github.com/notifications/unsubscribe-auth/AATvbJ0ApHk6caw-85ejuqnO_0V0-ZPoks5uoF-agaJpZM4Wfxyj .

[mcadam]

Hey, forgot to got back to you but looking good now so closing this one :)