Gradual fetching of data by scrolling down in File Manger Application

[mahsashadi]

According to this issue, I am implementing gradual fetching by scrolling in FM app. I have two issues.

1. I need to detect when the whole data is fetched to stop fetching the next time scroll hits the bottom. So I think I need to initially get total number of directories and files of a path (without calling readdir api and getting the list) and save it into some state. I checked the stat api, but it does not provide this attribute.
2. I need to have the last file returned from server, but the list returned here is not in the same order as server response, due to some sorting done here.

[mahsashadi]

BTW, What is your idea about adding this pagination feature to your file manager app? As you suggest here, it can be used by adapters supported this interface in their api. If you agree, I can make a pull request when it is totally done.

[andersevenrud]

Hm.

Relying on either the total number of files or the last one to figure out when to stop does not sound like a good idea.

An easier way to do this is simply to set a "page size" and whenever you get a "page" back from the api that is less than this page size or zero, you know you've reached the end.

[mahsashadi]

An easier way to do this is simply to set a "page size" and whenever you get a "page" back from the api that is less than this page size or zero, you know you've reached the end.

Yes, seems a better idea, thanks :)

but I still need to have the last file of each page, returned by server, as a token for getting the next page.

[andersevenrud]

but I still need to have the last file of each page, returned by server, as a token for getting the next page.

It is possible just to use a page number :thinking:

const pageSize = 10
let currentPage = 0
let reachedEnd = false

function changedDirectory() {
  currentPage = 0
  reachedEnd = false
}

async function scrollReachedBottom() {
  if (reachedEnd) return
  const nextPage = currentPage + 1
  const result = await readdir('home:/', { page: nextPage, pageSize })
  reachedEnd = result.length < pageSize
  currentPage = nextPage
}

Now you have a offset on the server: page * pageSize.

[mahsashadi]

But it seems my storage pagination API does not support offset. https://docs.openstack.org/swift/latest/api/pagination.html

[andersevenrud]

I see. I'm not really sure how to solve this then without also adding sorting as an option (default true).

[mahsashadi]

So do you think you can add this feature? Or if it is needed to be done by myself, I can start working on that.

[andersevenrud]

Feel free to have a look at it. This would also mean that the file manager would have to run the sort manually to make it behave the same.

[mahsashadi]

Great :) Do you have any idea which parts of codes are potential for changing and also where this boolean option should be set.

[andersevenrud]

I was just thinking of adding it to the options and add the logic to the VFS function:

https://github.com/os-js/osjs-client/blob/385b38f3c4b113d1afaf5775c8402e5f22436a96/src/vfs.js#L84

https://github.com/os-js/osjs-client/blob/385b38f3c4b113d1afaf5775c8402e5f22436a96/src/vfs.js#L70-L75

[mahsashadi]

This would also mean that the file manager would have to run the sort manually to make it behave the same.

About this part, when a new page is fetched, I think sorting the accumulative list (previous pages+ new page) leads to a bad UX. The files of previous pages and the new one might be mixed, and the user can not scroll just among files of new page.

[andersevenrud]

I was only thinking about applying sorting per result.

Because in adapters that does not support this you'd only get one result.

[mahsashadi]

So by adding sortFiles option to this method as below:

// Handles directory listing result(s)
const handleDirectoryList = (path, options) => result =>
  Promise.resolve(result.map(stat => createFileIter(stat)))
    .then(result => transformReaddir(pathToObject(path), result, {
      showHiddenFiles: options.showHiddenFiles !== false,
      filter: options.filter,
      sortFiles: options.sortFiles
    }));

We just need to check sortFiles option in transformReaddir method and do sorting if it is true. This way it works :)

The question is where should I config this boolean option? Maybe it is needed to add this config to src/client/config.js:

  fileManager:{
    pageSize: 1000,
    sortFiles: false
  }

[andersevenrud]

The question is where should I config this boolean option?

Doesn't this just fit right into the FM readdir function ?

[mahsashadi]

BTW, What is your idea about adding this pagination feature to your file manager app? As you suggest here, it can be used by adapters supported this interface in their api. If you agree, I can make a pull request when it is totally done.

So when we complete this feature, are you going to add it to your FM app? If so is it possible to set in readdir as below?

const options = {
  showHiddenFiles: proc.settings.showHiddenFiles,
  sortFiles: false,
  page:{
    size:1000,
    number: state.currentPage,
    lastFetchedFile: state.lastFetchedFile
  }
};

In this way when sorting (deafult is true) is going to happen?

[andersevenrud]

If so is it possible to set in readdir as below?

Yes, that will be possible. However, it would be nice to agree on a default option and properties for "paging" so that it can work for all adapters, not just the one that you're making.

[mahsashadi]

For sure we should agree on paging options. I've just only used some options for testing the functionality.

Up to now, Three options seem necessary in my mind:

1. size of each page
2. number of current page (for storage services that use offset)
3. last fetched file name (for storage services like our case)

[mahsashadi]

Up to now, Three options seem necessary in my mind:

So what is your opinion ?

[mahsashadi]

I again find some time to work on paging project :) So if you have any time, it's good to agree on paging properties. Thanks a lot.

[andersevenrud]

So if you have any time

Sadly, I've been busy with a lot of other things. Those three options are fine, but I can also thing of another one: "pagination token". This is for examplle have in Google Drive. Maybe there are other options there to consider as well.

[mahsashadi]

Sometimes I fetch some pages in File manager app, i faced the following error (the page list is returned correctly)

Do you know what can cause this error?

[andersevenrud]

Hm. Pretty sure that this can occur if you remove something from the VDOM while it's updating. Does it cause any issues ?

[mahsashadi]

It causes new page list items does not show on the file manager app.

[andersevenrud]

Can you share the code that causes this ?

[mahsashadi]

And there is also another question. By having pagination, I think it is better to initially show the total count of directories and files of each directory in statusbar ( not increasing by list updating whenever a new page is fetched)

If you agree, what is your suggestion for its implementation?

[mahsashadi]

Can you share the code that causes this ?

I do not have access right now. I will share it tomorrow 🙏

[andersevenrud]

By having pagination, I think it is better to initially show the total count of directories and files of each directory in statusbar ( not increasing by list updating whenever a new page is fetched)

This would require getting some additional information, which is now not really possible in the VFS :thinking:

I'm starting to think maybe having a new scandir endpoint that is designed for doing things like this so that you get a proper server response:

{
  count: number, // Files in this response (page)
  totalFiles: number, // Total files in the directory
  totalPages: number, // Total number of pages
  currentPage: number | string, // An identifier that is used to identify the current page
  next?: {}, // Options for scandir to give next page ?
  prev?: {}, // Options to scandir give previois page ?
  data: VFSFile[]
}

I do not have access right now. I will share it tomorrow pray

Great!

[mahsashadi]

How about providing number of files and directories of each dir in stat response.

So we should call stat whenever a new mountpoint or directory is selected in app.

[andersevenrud]

Oh, yeah. I didn't think of that :man_facepalming: That could work!

[mahsashadi]

These are changes in filemanger application index file (till now):

const PAGE_SIZE = 10;

/**
 * Update some state properties when selected directory/file changed
 */
const updateState = (state) => {
  state.currentList = [];
  state.currentPage = 0;
  state.fetchAllPages = false;
  state.currentLastItem = '';
};

/**
 * Detect pagination capability when selected mountpoint changed
 */
const isPagingActive = async (core, path) => {
  const vfs = core.make('osjs/vfs');
  const capabilities = await vfs.capabilities(path);
  return capabilities.pagination;
};

/**
 * Create files list by concating new page items by previous fetched pages items
 */
const createPagesList = async (proc, state, vfs, dir) => {
  const options = {
    showHiddenFiles: proc.settings.showHiddenFiles,
    page:{
      size:PAGE_SIZE,
      number: state.currentPage,
      marker: state.currentLastItem,
      token: ''
    }
  };

  let list = await vfs.readdir(dir, options);
  // FIXME: the special file `..` should be handled in a better way (not add per page).
  if(list.length === 0 || (list.length === 1 && list[0].filename === '..')) {
    state.fetchAllPages = true;
  }
  if(list.length !== 0 && list !== state.currentList) {
    // FIXME: the special file `..` should be handled in a better way (not add per page).
    if(state.currentList.length !== 0 && list[0].filename === '..' && state.currentList[0].filename === '..') {
      list = list.filter(item => item.filename !== '..');
    }
    state.currentLastItem = list.length !== 0 ? list[list.length - 1].filename : null;
    state.currentList = state.currentList.concat(list);
    list = state.currentList;
  } else {
    list = state.currentList;
  }
  return list;
};

/**
 * Create total list of files when pagination is not supported
 */
const createTotalList = (proc, vfs, dir) => {
  const options = {
    showHiddenFiles: proc.settings.showHiddenFiles
  };
  return vfs.readdir(dir, options);
};

/**
 * VFS action Factory
 */
const vfsActionFactory = (core, proc, win, dialog, state) => {
  // ...
  const readdir = async (dir, history, selectFile) => {
    if (win.getState('loading')) {
      return;
    }
    // if calling by scroll
    if(dir === undefined) {
      dir = state.currentPath;
      state.currentPage += 1;
    }

    try {
      const message = __('LBL_LOADING', dir.path);
      win.setState('loading', true);
      win.emit('filemanager:status', message);
      let list;
      if(state.pagination) {
        list = await createPagesList(proc, state, vfs, dir);
      }else {
        list = await createTotalList(proc, vfs, dir);
      }
    // ...
};

/**
 * Creates a new FileManager user interface
 */
const createApplication = (core, proc, state) => {
   const createActions = (win) => ({
     // ...

    mountview: listView.actions({
      select: async ({data}) => {
        await updateState(state);
        state.pagination = await isPagingActive(core, data.root);
        win.emit('filemanager:navigate', {path: data.root});
      }
    }),

    fileview: listView.actions({
        // ...
        activate: async ({data}) => {
             await updateState(state);
             win.emit(`filemanager:${data.isFile ? 'open' : 'navigate'}`, data);
      },
        scroll: (ev) => {
        if (state.pagination) {
          const el = ev.target;
          if (state.fetchAllPages) {
            return;
          }
          const hitBottom = (el.scrollTop + el.offsetHeight) >= el.scrollHeight;
          if(hitBottom) {
            win.emit('filemanager:navigate');
          }
        }
      }
)}
 // ...
}

/**
 * Creates a new FileManager window
 */
const createWindow = async (core, proc) => {
  let wired;
  const state = {
    currentFile: undefined,
    currentPath: undefined,
    currentList: [],
    currentPage:0,
    fetchAllPages: false,
    currentLastItem:'',
    pagination: false
  };
  const {homePath, initialPath} = createInitialPaths(core, proc);
  state.pagination = await isPagingActive(core, initialPath);
  // ...
}

And I changed transformReaddir of src/utils/vfs.js as below:

 const sortedSpecial = createSpecials(path)
    .sort(sorter(sortBy, sortDir))
    .map(modify);

  const sortFiles = files.filter(filterHidden)
    .filter(filter)
    .map(modify);

  return [
    ...sortedSpecial,
    ...sortFiles
  ];

[mahsashadi]

When that error accurs:

• New page items in current directory are not shown
• By transferring between directories/mountpoints, as shown in pic, it is not truly rendering and last files from previous routed directory are shown in current directory (folder14 and folde22 are not in home but shown!)

[andersevenrud]

I think I'll have to actually run these changes myself to get a better understanding. Do you have this in a repo ?

[mahsashadi]

Pretty sure that this can occur if you remove something from the VDOM while it's updating

I worked more on that.

This error happened whenever there is at least one duplicate in array list. So It can be solved by filtering the array as below:

  const filenameSet = new Set();
  const filteredArr = list.filter((obj) => {
    const isPresentInSet = filenameSet.has(obj.filename);
    filenameSet.add(obj.filename);
    return !isPresentInSet;
  });

[mahsashadi]

What is your suggestion for the page-size and number of pages can be shown on filemanager?

• Option A: show as much as pages that are fetched ( :-1: not seems as a good idea, scrolling among huge amount of files will be hard)
• Option B: only one page can be shown to the user, means that by fetching new page, the previous page items will be deleted (maybe good, it depends on how we set our page-size)
• Option C: we can set a limit for number of pages, for example maximum 5 pages items can be listed.
  • when page 6 is fetched, page 1 items will be deleted.
  • when page 7 is fetched, page 2 items will be deleted.
  • and so on ... .

Option C seems better, but its implementation can be a bit complicated, since filesystems work differently (in my case it works by marker not offset)

[mahsashadi]

By having pagination, I think it is better to initially show the total count of directories and files of each directory in statusbar ( not increasing by list updating whenever a new page is fetched)

current statusbar: 20 directories, 10 files, 100 bytes total

new statusbar (option 1): 20/100 directories, 10/200 files, 100/10000 bytes total

new statusbar (option 2): 30/300 files & directories, 100/10000 bytes total

Which option do you think is better? if option 1 is needed, is it possible to return count of dir and files seperately in stat?

for option 2, I changed this code block as below:

 const createStat = async stat => {
    let count = null;
    if (stat.isDirectory()) {
      const files = await fs.readdir(realPath);
      count = files.length;
    }
    return ({
      isDirectory: stat.isDirectory(),
      isFile: stat.isFile(),
      mime: stat.isFile() ? mime(realPath) : null,
      size: stat.size,
      path: file,
      count: count,
      filename,
      stat
    });
  };

[andersevenrud]

Hm. I think maybe it would be best if it simply said x of 100 files, 100000 bytes total. Look kind of confusing with all those stats in there :thinking:

is it possible to return count of dir and files seperately in stat?

Well, only if the filesystem that you're working with supports it.

[mahsashadi]

x of 100 files, 100000 bytes total

So by files you mean files and directories and x in x bytes total show current loaded bytes (by fetched pages), true?

[mahsashadi]

for option 2, I changed this code block as below:

And what should I do for hidden files? Should I also filter them here server side to have total count without hidden files?

[andersevenrud]

With x of 100 files, 100000 bytes total I meant that x was the current loaded count (from fetched pages), and everything behind that was the actual total count.

Should I also filter them here server side to have total count without hidden files?

Hidden files should probably only be counted if the option to show them is enabled.

[mahsashadi]

With x of 100 files, 100000 bytes total I meant that x was the current loaded count (from fetched pages), and everything behind that was the actual total count.

Actually my question was about the number of total bytes, which I guess you mean there is no need to show current bytes, just show total bytes.

What is your suggestion for the page-size and number of pages can be shown on filemanager?

So do you have any suggestion for this functionality?

[andersevenrud]

which I guess you mean there is no need to show current bytes, just show total bytes.

Yes, the total count.

So do you have any suggestion for this functionality?

Does this even need to be exposed ? If it says x of y files that should be enough.

As for adjusting the page size, this is probably better as an adapter setting, and not something that you change in the UI.

[mahsashadi]

Does this even need to be exposed ? If it says x of y files that should be enough

I did not understand this. I previously mentioned these three options.

• Option A: show as much as pages that are fetched ( -1 not seems as a good idea, scrolling among huge amount of files will be hard)
• Option B: only one page can be shown to the user, means that by fetching new page, the previous page items will be deleted (maybe good, it depends on how we set our page-size)
• Option C: we can set a limit for number of pages, for example maximum 5 pages items can be listed.

So by considering x of 100 files, 100000 bytes total format for statusbar, I did not understand which option is good in your idea?

[andersevenrud]

I did not understand which option is good in your idea?

None of the options.

There is not any need if the user already sees x of 100 files, 100000 bytes total. The user should not really know about how many pages etc there is, only the total count.

[mahsashadi]

There is not any need if the user already sees x of 100 files, 100000 bytes total. The user should not really know about how many pages etc there is, only the total count.

I did not mean to show number of pages somewhere, Actually I am thinking about list of page items (files) itself. Consider situation when 4-5 pages with pageSize 1000 fetched. Scrolling among lots of files might not be user-friendly. I was thinking about listing items in a way that solves this situation (as option A, B, C)

[andersevenrud]

Scrolling among lots of files might not be user-friendly.

I don't really think there's a good way to solve this outside maybe doing some kind of categorization/grouping, which would not even be possible with pagination. Option B is very bad UX and option C will probably make it so that the user can't view all of their files, right ?

[mahsashadi]

So your opinion is that it is better to show as many pages as fetched, right?

option C will probably make it so that the user can't view all of their files, right ?

yes, maybe previous files fetched again by scrolling up!

[andersevenrud]

So your opinion is that it is better to show as many pages as fetched, right?

Yes, show everything.

yes, maybe previous files fetched again by scrolling up!

Hm. I'm not sure how this would be implemented, or how it would "feel" (UX) :thinking:

[mahsashadi]

Hi :)

What is your suggestion for refreshing list to support actions like delete, paste and rename while pagination is active.

Imagine we have deleted a file within one of our previous fetched pages.