Nuttx file descriptor dup operation issue

[Donny9]

Issue

The dup operation of nuttx is different from that of freebsd and opengroup.

• Nuttx The old file descriptor and the new file descriptor generated by dup and dup2 don't share the file entity and are independent of each other. They correspond to different struct file entities in the filelist.

  A file descriptor is an index into an fl_files array of sruct file. The struct file associates the file descriptor to the file state and to a set of inode operations.

  struct file
  {
  int               f_oflags;   /* Open mode flags */
  off_t             f_pos;      /* File position */
  FAR struct inode *f_inode;    /* Driver or file system interface */
  FAR void         *f_priv;     /* Per file driver private data */
  };
  
  /* This defines a two layer array of files indexed by the file descriptor.
  * Each row of this array is fixed size: CONFIG_NFILE_DESCRIPTORS_PER_BLOCK.
  * You can get file instance in filelist by the follow methods:
  * (file descriptor / CONFIG_NFILE_DESCRIPTORS_PER_BLOCK) as row index and
  * (file descriptor % CONFIG_NFILE_DESCRIPTORS_PER_BLOCK) as column index.
  */
  
  struct filelist
  {
  sem_t             fl_sem;     /* Manage access to the file list */
  uint8_t           fl_rows;    /* The number of rows of fl_files array */
  FAR struct file **fl_files;   /* The pointer of two layer file descriptors array */
  };                                                                                   

• FreeBSD/OpenGroup:

• Do you have good idea to fix this issue? In view of the above problems, it will cause some problems in the application of using dup, dup2 and fcntl on nuttx. Here is one of the methods: I will use filenode to correspond to file descriptor one-to-one, and multiple filenodes can correspond to a unified file entity, When dup occurs, the file pointer corresponding to the old filenode can be copied to the new filenode to ensure sharing.

  
  struct file
  {
  int16_t           f_crefs;    /* References to file */
  int               f_oflags;   /* Open mode flags */
  off_t             f_pos;      /* File position */
  FAR struct inode *f_inode;    /* Driver or file system interface */
  FAR void         *f_priv;     /* Per file driver private data */
  };

/* A file descriptor is an index into an array of such types. The type is used to

• multiple file descriptor to share the same open file, include file offset and
• file status flags, buf don't share file descriptor flags "close-on-exec flag". */

struct filenode { int fn_oflags; / Open or duplicate mode flags / FAR struct file fn_file; / The pointer representation of an open file */ };

/* This defines a two layer array of files indexed by the file descriptor.

• Each row of this array is fixed size: CONFIG_NFILE_DESCRIPTORS_PER_BLOCK.
• You can get file instance in filelist by the follow methods:
• (file descriptor / CONFIG_NFILE_DESCRIPTORS_PER_BLOCK) as row index and
• (file descriptor % CONFIG_NFILE_DESCRIPTORS_PER_BLOCK) as column index. */

struct filelist { sem_t fl_sem; / Manage access to the file list / uint8_t fl_rows; / The number of rows of fl_files array / FAR struct filenode *fl_files; / The pointer of two layer file descriptors array */ };

And it also needs to modify file_openation:dup, directly delete or remove unnecessary features.

# Refer to
https://www.freebsd.org/cgi/man.cgi?query=dup&sektion=2
https://pubs.opengroup.org/onlinepubs/009604499/functions/dup.html

[xiaoxiang781216]

@pkarashchenko @patacongo @acassis @masayuki2009 what do you think about this bug?

[pkarashchenko]

I see that FreeBSD indeed includes such description, but I failed to find any similar lines in https://pubs.opengroup.org/onlinepubs/9699919799/functions/dup.html

Additionally I can find in description from OpenGroup: The dup() function provides an alternative interface to the service provided by fcntl() using the F_DUPFD command. The call dup(fildes) shall be equivalent to:

fcntl(fildes, F_DUPFD, 0);

But fail to find similar in description from FreeBSD

[Donny9]

@pkarashchenko

I see that FreeBSD indeed includes such description, but I failed to find any similar lines in https://pubs.opengroup.org/onlinepubs/9699919799/functions/dup.html

Additionally I can find in description from OpenGroup: The dup() function provides an alternative interface to the service provided by fcntl() using the F_DUPFD command. The call dup(fildes) shall be equivalent to:

fcntl(fildes, F_DUPFD, 0);

But fail to find similar in description from FreeBSD

@pkarashchenko FreeBsd also have similar behavior in desctiption: https://man.freebsd.org/cgi/man.cgi?query=dup2&apropos=0&sektion=2&manpath=FreeBSD+9.0-RELEASE&arch=amd64&format=html#:~:text=The%20object%20referenced,between%20the%20references.

The object referenced by the descriptor does not distinguish between oldd
and newd in any way.  Thus if newd and oldd are duplicate references to
an open file, [read(2)],  [write(2)] and [lseek(2)] calls all move a single
pointer into the file, and append mode, non-blocking I/O and asynchronous
I/O options are shared between the references.

So any good suggestions to solve this problem? @pkarashchenko @acassis If there are no good suggestions, I will use filenode to solve this problem in the near future

[pkarashchenko]

@Donny9 It's a bit find out from a description what is the issue from POSIX perspective, but finally I think I managed to understand what you are talking about. It is https://pubs.opengroup.org/onlinepubs/9699919799/functions/dup.html

The dup2() function shall cause the file descriptor fildes2 to refer to the same open file description as the file descriptor
fildes and to share any locks, and shall return fildes2.

Is that correct? If yes, then it depends on the interpretation if refer to the same open file description is equivalent to does not distinguish between oldd and newd in any way.

If I still didn't understood the issue correctly then please explain it in more clear manner like:

• state the POSIX description lines
• state FreeBSD description lines (that I think is already done)
• show how current NuttX implementation violates POSIX

[Donny9]

@pkarashchenko This examples can run with pass result on linux, but can't pass on nuttx, because this issue: newd and oldd can't share file offset, because they are different file handle in nuttx.

//#include <nuttx/config.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
#include <syslog.h>
//#include "test.h"
#define TESTFILE "testDup2File1"
#define TEST_FAIL -1
#define TEST_PASS 0

/****************************************************************************
 * Name: dup
 * Example description:
    1. Open a file and get the file identifier fd.
    2. Copy the file identifier to newfd
    3. Open the file through newfd. And check the return results
 * Test item: open()  write() dup2()
 * Expect results: TEST PASSED
 ****************************************************************************/

int main(void)
{
    char buf[16] = {0};

    off_t currpos;
    int fd1 = open(TESTFILE, O_RDWR | O_CREAT, 0777);
    if (fd1 == -1)
    {
        printf("open file fail !\n");
        return TEST_FAIL;
    }

    int fd2 = open(TESTFILE, O_RDWR | O_CREAT, 0777);
    if (fd2 == -1)
    {
        close(fd1);
        printf("open file fail !\n");
        return TEST_FAIL;
    }

    int ret = dup2(fd1, fd2);
    if (ret == -1)
    {
        close(fd1);
        close(fd2);
        printf("dup2 error ,  return : %d!\n", ret);
        return TEST_FAIL;
    }
    syslog(LOG_INFO, "current fd = %d\n", ret);
    char *buf1 = "hello ";
    char *buf2 = "world!";
    write(fd1, buf1, strlen(buf1));
    write(fd2, buf2, strlen(buf2));

    fsync(fd1);

    /* reset file pos use fd2 */
    lseek(fd2, 0, SEEK_SET);

    /* Check if file pos is shared */
    currpos = lseek(fd1, 0, SEEK_CUR);
    if (currpos != 0)
    {
        printf("Do not shared file pos after dup2()\n");
        close(fd1);
        close(fd2);
        return TEST_FAIL;
    }

    ret = read(fd1, buf, 16);
    if (ret <= 0)
    {
        printf("read fail\n");
        close(fd1);
        close(fd2);
        return TEST_FAIL;
    }
    close(fd1);
    close(fd2);

    if (strncmp(buf, "hello world!", 12) == 0)
    {
        printf("test pass\n");
        return TEST_PASS;
    }

    printf("test fail\n");
    return TEST_FAIL;
}

[pkarashchenko]

@Donny9 what about other OSes? I will try to run it on MAC OS and come back with results

[pkarashchenko]

The MACOS works the same and test pass. On your proposal I do not fully understand why struct filenode is needed. Could you please explain more. I see that struct filenode contains fn_oflags and struct file contains f_oflags, so what is the difference between those two?

[Donny9]

xplain more. I see that struct filenode contains fn_oflags and struct file contains f_oflags, so what is the difference between those two?

oldd and newd need to share file offsets and other information, but they need to have their own flags. We can remove the flags in the file structure, but it will change a lot. Therefore, we need to make oldd and newd point to the same file structure, so let the file be one level below and use filenode as the file opening context.

[pkarashchenko]

But leaving flags in both file and filenode structures will lead to a duplication? Or some flags will stay in file while other will reside in filenode? BTW I think that there is a typo in your test example. I didn't try to run it on NuttX, but believe that line ret = read(fd1, buf, 16); should be ret = read(ret, buf, 16); otherwise I do not see where dup2 result is used