dokan-dev / dokany

User mode file system library for windows with FUSE Wrapper
http://dokan-dev.github.io
5.27k stars 665 forks source link

Proposed IOCTL callback extension #544

Open jimuk opened 7 years ago

jimuk commented 7 years ago

I have just started to explore the Dokan project so please excuse me if this suggestion has been made before.

It would be greatl if the DOKAN_OPERATIONS structure was extended to provide an optional callback method for passing arbitrary IOCTL messages to the underlying file system code. For instance:

NTSTATUS(* DeviceIoControl )(LPCWSTR FileName, DWORD dwIoControlCode, LPVOID lpInBuffer, DWORD nInBufferSize, LPVOID lpOutBuffer, DWORD nOutBufferSize, LPDWORD lpBytesReturned, PDOKAN_FILE_INFO DokanFileInfo)

I would suggest this could have several uses/advantages:

a. Allow arbitrary data to be "tunneled" to the underlying file system code

b. Provide a way to implement application specific features without changing Dorkan.

c. Implement this in a Windows compatible way

Jim

Liryna commented 7 years ago

Hi @jimuk ,

Dokan is made to create filesystem. How IOCTL callback extension would help to do this ?

jimuk commented 7 years ago

Hi @Liryna,

Dorkan is great and has a very clear and straight forward API. This allows almost any information to be represented as a Windows friendly file system and explored/managed using familiar tools like Explorer.

However, the flip side of this simplicity is that there are a very limited number of ways to interact with a file and it's metadata. For instance, SetFileTime() supports exactly 3 timestamps but some file systems may have more (NTFS has 4 or more if you could additional "hidden" metadata).

The IOCTL interface would would allow Dorkan to be used as the basis for more complex integrations. At a basic level information for the underlying system (think SQL or another "alien" file system) would appear as a regular Windows file system but applications on top of this layer could use the IOCTL mechanism to query additional features and metadata by "tunneling" this information through Dorkan.

I think I could probably write a patch to add this feature but before I started on this I wanted to check if it t had been discussed before and if it was likely to be accepted by the community.

I hope this helps.

Jim

Rondom commented 7 years ago

I don't have any clue about Windows programming. What is the difference between IOCTLs and FSCTLs and why would one use one over the other?

jimuk commented 7 years ago

I don't think there is much practical difference between them. I understand that FSCTL codes are used to communicate with file system drivers whilst IOCTL are more general and used to communicate with any driver. Basically, they are a way for non-data IO with the underlying driver. The method call for the IOCTL allows a buffer to be passed down to the driver and a buffer received back. Both are optional and the communication can be one way if require or even pass no data at all. The Linux ioctl() works in a similar way.

Jim

Rondom commented 7 years ago

I think it sounds good. @Liryna, what do you think? I think it could be useful to support both IOCTLs and FSCTLs if it is not too much extra work.

Liryna commented 7 years ago

This can be a good idea if it is asked by the community 👍 @jimuk I do not know how you would like to implement it but I would be really great to no break current kernel <-> library communication or otherwise this will need to be moved to the 2.x.x version.

jimuk commented 7 years ago

If I understand the Dokan API it will require an extra entry in the DOKAN_OPERATIONS structure. Would this break the kernel <-> library communication?

BTW: I haven't read about future plans for Dokan but maybe the structure could be replaced by providing functions to "register" handlers with the driver code. This would make it more extensible in the future and provide some backwards compatibility with older drivers?

Jim

marinkobabic commented 7 years ago

The question is about how many additional IOCTL are we talking here which are actually not implement in Dokan? Depending on how big the list is it may make sense just to implement them. Don't think that a file system changes the IOCTL that much in future.

Can you provide such a list please?

Am 13.07.2017 23:04 schrieb "jimuk" notifications@github.com:

If I understand the Dokan API it will require an extra entry in the DOKAN_OPERATIONS structure. Would this break the kernel <-> library communication?

BTW: I haven't read about future plans for Dokan but maybe the structure could be replaced by providing functions to "register" handlers with the driver code. This would make it more extensible in the future and provide some backwards compatibility with older drivers?

Jim

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dokan-dev/dokany/issues/544#issuecomment-315201942, or mute the thread https://github.com/notifications/unsubscribe-auth/ADHSOENsJazm-EbHx6tkVkPgJM6yDlbAks5sNoZigaJpZM4OKc5o .

magol commented 7 years ago

I think this is probably a good idea. The file system that we implement is quite different from NTFS. There are many metadata that we can not convey through the file system, but that we really want to use. If it had been great to solve this in some way.

And I think it is a great idea to get rid of DOKAN_OPERATIONS to next version and replace it with any kind of registration as @jimuk suggests. As it is now, it is not possible to add any new functionality without destroy backward compatibility.

jimuk commented 7 years ago

To answer the points made by @marinkobabic and @Rondom. This change only requires one new callback handler adding to driver. This is a generic mechanism that any IOCTL/FSCTL (they are essentially the same) can be tunneled through. The IOCTL mechanism supports three modes of operation:

  1. Read - Pull data from underlying driver
  2. Write - Push data down to underlying driver
  3. Read-Write - Simultaneous of both above

The following MSDN page explains how Windows uses this:

[(https://msdn.microsoft.com/en-us/library/windows/desktop/aa363219(v=vs.85).aspx)]

Dokan does not yet support async IO so I would suggest we drop the lpOverlapped member. This would leave the following handler signature:

NTSTATUS(* DeviceIoControl )(LPCWSTR FileName, DWORD dwIoControlCode, LPVOID lpInBuffer, DWORD nInBufferSize, LPVOID lpOutBuffer, DWORD nOutBufferSize, LPDWORD lpBytesReturned, PDOKAN_FILE_INFO DokanFileInfo)

Jim

marinkobabic commented 7 years ago

What happens when you use one of the already implemented IOCTL by Dokan?

What IOCTL implementations are you missing actually?

You request is in generally: please provide me a way in user Mode to talk directly to the operating system.

Am 14.07.2017 10:39 schrieb "jimuk" notifications@github.com:

To answer the points made by @marinkobabic https://github.com/marinkobabic and @Rondom https://github.com/rondom. This change only requires one new callback handler adding to driver. This is a generic mechanism that any IOCTL/FSCTL (they are essentially the same) can be tunneled through. The IOCTL mechanism supports three modes of operation:

  1. Read - Pull data from underlying driver
  2. Write - Push data down to underlying driver
  3. Read-Write - Simultaneous of both above

The following MSDN page explains how Windows uses this:

[(https://msdn.microsoft.com/en-us/library/windows/desktop/ aa363219(v=vs.85).aspx)]

Dokan does not yet support async IO so I would suggest we drop the lpOverlapped member. This would leave the following handler signature:

NTSTATUS(* DeviceIoControl )(LPCWSTR FileName, DWORD dwIoControlCode, LPVOID lpInBuffer, DWORD nInBufferSize, LPVOID lpOutBuffer, DWORD nOutBufferSize, LPDWORD lpBytesReturned, PDOKAN_FILE_INFO DokanFileInfo)

Jim

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dokan-dev/dokany/issues/544#issuecomment-315305208, or mute the thread https://github.com/notifications/unsubscribe-auth/ADHSOPUVLDAVFMyVlRsdOjbebq0SSZssks5sNylWgaJpZM4OKc5o .

jimuk commented 7 years ago
  1. Example user mode code to fetch an arbitrary 100 bytes from the underlying driver could be:
CHAR cBuffer[100];
DWORD dwBytesRead;

BOOL bResult=WINAPI DeviceIoControl(hDevice, MY_CONTROL_CODE, cBuffer, 100, NULL, 0, &dwBytesRead, NULL);
  1. Dokan would tunnel this down to the kernel mode driver and then back to the user mode file system code. It would be passed to a method like:

NTSTATUS ProcessDeviceIoControl(LPCWSTR FileName, DWORD dwIoControlCode, LPVOID lpInBuffer, DWORD nInBufferSize, LPVOID lpOutBuffer, DWORD nOutBufferSize, LPDWORD lpBytesReturned, PDOKAN_FILE_INFO DokanFileInfo)

  1. This method would resolve the dwIoControl code, perform the desired operation and return the requested bytes.

Jim

marinkobabic commented 7 years ago

This means you are here talking always about custom IOCTL codes and not about the official available codes?

Am 14.07.2017 18:48 schrieb "jimuk" notifications@github.com:

  1. Example user mode code to fetch an arbitrary 100 bytes from the underlying driver could be:

`CHAR cBuffer[100]; DWORD dwBytesRead;

BOOL bResult=WINAPI DeviceIoControl(hDevice, MY_CONTROL_CODE, cBuffer, 100, NULL, 0, &dwBytesRead, NULL); `

  1. Dokan would tunnel this down to the kernel mode driver and then back to the user mode file system code. It would be passed to a method like:

NTSTATUS ProcessDeviceIoControl(LPCWSTR FileName, DWORD dwIoControlCode, LPVOID lpInBuffer, DWORD nInBufferSize, LPVOID lpOutBuffer, DWORD nOutBufferSize, LPDWORD lpBytesReturned, PDOKAN_FILE_INFO DokanFileInfo)

  1. This method would resolve the dwIoControl code, perform the desired operation and return the requested bytes.

Jim

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dokan-dev/dokany/issues/544#issuecomment-315408706, or mute the thread https://github.com/notifications/unsubscribe-auth/ADHSOAGeAhK1Pk_DRJuyghXyG1i2prVOks5sN5vtgaJpZM4OKc5o .

jimuk commented 7 years ago

The mechanism is generic. It can be used to pass any data in either direction from user app to user mode file system driver. It would provide a generic means to implement extension functionality without rebuilding Dolan each time.

Jim

marinkobabic commented 7 years ago

How we can prevent conflicts with existing implementation of Dokan? What happens when you send IOCTL for file writes or oplock?

Am 14.07.2017 19:48 schrieb "jimuk" notifications@github.com:

The mechanism is generic. It can be used to pass any data in either direction from user app to user mode file system driver. It would provide a generic means to implement extension functionality without rebuilding Dolan each time.

Jim

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dokan-dev/dokany/issues/544#issuecomment-315422976, or mute the thread https://github.com/notifications/unsubscribe-auth/ADHSOHSnCeEYQvnlQzb95vSKJmqM8_6Bks5sN6n9gaJpZM4OKc5o .

jimuk commented 7 years ago

I don't think there is any conflict. IOCTL is independent of normal data read/write. Typically it would be used for non-data requests lime retirving metadata but that is entirely up to implementtation. All Dorkan should do (like Windows) is pass the requests back and forth. It doesn't need to interpret

Jim

marinkobabic commented 7 years ago

If it works the way you explained then it would be a nice feature 😁

Am 14.07.2017 22:56 schrieb "jimuk" notifications@github.com:

I don't think there is any conflict. IOCTL is independent of normal data read/write. Typically it would be used for non-data requests lime retirving metadata but that is entirely up to implementtation. All Dorkan should do (like Windows) is pass the requests back and forth. It doesn't need to interpret

Jim

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dokan-dev/dokany/issues/544#issuecomment-315465694, or mute the thread https://github.com/notifications/unsubscribe-auth/ADHSOFJFTOG2_aPfW0Na_jc-qhq5JyaBks5sN9X0gaJpZM4OKc5o .

Maxhy commented 7 years ago

At first sight it could be a nice feature as I personally already needed it to perform driver specific features.

But I'm asking myself about the following points, to discuss:

jimuk commented 7 years ago

In response to @Maxhy's points I would make the following comments:

  1. There may be other ways to communicate from user mode to the file system code but IOCTL is the standard way this is done in both Windows and Linux. It therefore represents the most universal way to provide this additional functionality.

  2. IOCTL would not kill the abstraction layer. It provides a generic mechanism to "tunnel" application specific data over the abstraction layer. Windows uses the same model to "layer" driver and user mode code. Each Windows component can listen for specific IOCTL codes and, if appropriate, respond to them. In all other cases they are just passed down/up to the next layer. Again, this is therefore a universal approach. Similarly, Dokan doesn't need to understand the IOCTL packets to pass them up/down. The mechanism is truly generic.

  3. Dokan will be able to validate the IOCTL packet buffer address and size. It can therefore check that the IOCTL mechanism is being called correctly and avoid BSOD issues in the driver. The only thing it won't be able to do is validate the contents of the IOCTL packet. However, this shouldn't be a concern this is an application specific detail that Dokan cannot be expected to know about.

Rondom commented 7 years ago

I think it is important to distinguish between custom IOCTLs/FSCTLs and Windows ones. The latter should probably be handled within the kernel driver itself, to keep things simple and keep some level of abstraction. What do you think?

How would one distinguish between the two? Is there a range that Windows reserves for third-party code? That way, we could basically say:

if (custom_ioctl)
    pass_to_userspace()
else {
    // handle windows  ioctls
}
jimuk commented 7 years ago

There really is nothing special about Windows IOCTL codes. They are relevant to specific Windows components but otherwise just like any other IOCTL to everything else. I would therefore suggest a slightly different approach:

a. Pass IOCTL codes through up/down the driver (e.g. in / out buffer)

b. Support specific IOCTL codes inside Dokan if relevant (for instance IOCTL_DISK_GET_DRIVE_GEOMETRY)

atanamir commented 6 years ago

I'd like to +1 this even though it's been inactive for some time. And reading through the whole thread, it's definitely a 2.0 feature because it would change the kernel <-> library implementation. We'd have to filter out the known (windows) FSCTLs/IOCTL codes in the kernel, but only pass unrecognized ones to the dokan user-mode process to complete.

In our case, we'd also like to use this for applications to request additional non-standard metadata from our dokan-based file system.

jimuk commented 6 years ago

There is no need to filter out known FSCTLS/IOCTLs. Windows uses an IO stack model which allows all unhandled control codes to filter down/up to the next layer - Basically, if you care you handle it. If you don't care / don't know you should ignore it.

I'm also not sure why this feature would need to wait to v2.0? It will require a change to the DOKAN_OPERATIONS structure but one additional callback will be sufficient to handle all control codes because the mechanism is generic and infinitely extensible.

Jim

Liryna commented 6 years ago

Changing the DOKAN_OPERATIONS structure would break the API, so we would have to change the major version. https://semver.org/

magol commented 6 years ago

As @Liryna said, it is unfortunately not possible to extend functionality without breaking backward compatibility.

It's a problem, I'd love to see that we somehow change this model to version 2 so we can expand with new features in the future without breaking compability (se Open/closed principle). What it would look like, I do not know. Maybe create a new issue for that discusion?

jimuk commented 6 years ago

I understand that it would change the API and that this should best be done when introducing a new major version number. This would avoid any problems with backwards compatibility.

In the future, instead of using the DOKAN_OPERATIONS structure, a different solution would be to export a "registration" method from the DLL to register the callback/handler for each library operation such as read, write, IOCTL and so on. This would allow the interface to be extensible in the future and avoid the problems with the structure.

Jim

magol commented 6 years ago

@jimuk I agree with you. As long as the performance is not affected, I think it could be a good solution.

Rondom commented 6 years ago

A common pattern is to leave some reserved members at the end of the struct and instruct callers to keep them zero. Adding a new member does not break compatibility.

More extensible and also enabling providing ABI compatibility is using a version macro, i.e. users of the library define the version of the library that they desire. The header then defines the structs and the functions taking those structs according to the version. New versions still support the old version by converting the arguments accordingly. This is what I had planned to enable at least source-level compatibility for Dokan 1 => 2.

(I hope what I wrote is understandable.)

kyanha commented 4 years ago

I would like to +1 this. But I don't think the Windows fsctl codes should be filtered out -- one of the main reasons why we can't do a lot of the cool things that native filesystems can do is because we don't receive the Windows-defined fsctl codes (such things as "get reparse point", "set reparse point", "get file ID", "set file ID", "recall from remote storage", "encrypt file with EFS", "decrypt file with EFS", "zero a region of a sparse file", etc).

Being able to receive and dispatch those fsctls does also require being able to tell Windows that those capabilities are available to call from the filesystem.

glenn-slayden commented 2 years ago

I agree with @kyanha; I was initially excited to discover (and learn about) dokan(y), but I was assuming my custom filesystem would be able to emit NTFS reparse points, the small files that would need to be have a REPARSE_TAG attached (These would be simulated on-the-fly by my dokany FS to redirect the IO request back into an actual NTFS volume). It's a showstopper if I can't do this with your (otherwise entirely alluring) dokany project.

Liryna commented 2 years ago

I understand the purpose of this request and I am not against having it but this is a niche feature which I will personally not have time to implement. The hope is someone that expect to be able to implement correctly the ioctl in user land would also have the knowledge to contribute to the project and add the feature on kernel side 🚀 😎

kyanha commented 2 years ago

One of the reasons folks use Dokany is because they're not set up for kernel-mode development. I might technically have the knowledge to be able to contribute to the kernel (even though I'm kind of terrified by the idea), but I don't have a setup which would allow me to do so without crashing my coding machine.

Are there good guides for this? Can I set up a test machibe in a VM? (I prefer using VirtualBox to Hyper-V, if that's a concern.)

kyanha commented 2 years ago

One of the reasons folks use Dokany is because they're not set up for kernel-mode development. I might technically have the knowledge to be able to contribute to the kernel (even though I'm kind of terrified by the idea), but I don't have a setup which would allow me to do so without crashing my coding machine.

Are there good guides for this? Can I set up a test machibe in a VM? (I prefer using VirtualBox to Hyper-V, if that's a concern)

Liryna commented 2 years ago

I agree that getting the setup is not that easy but it is documented here https://github.com/dokan-dev/dokany/wiki/Build https://github.com/dokan-dev/dokany/wiki/How-to-Debug-Dokan If something is missing or someone have a question, feel free to ping me!