[SYCL][SPEC] USM regarding to context

[CaoZhongZ]

I make statement, anything wrong please tell me.

I want to understand USM allocation regarding to "device" or "context". Consider Unified Addressing, Virtual Address Space tied to a context, consider "shared" part, real resource backing the virtual address is tied to device.

The problem is when Unified Addressing tied to multiple contexts in a single process. So the same virtual address with different context belongs to different virtual address space and are totally different entities, could point to completely different physical resources?

[bader]

According to my understanding your statement is correct. +@jbrodman to confirm.

[jbrodman]

Yes.

Virtual addresses are not guaranteed to have any sort of consistency across different devices.

[CaoZhongZ]

I brought up this to attention because if a unique virtual address could point to at least two different context resource, when migration happens it'll be very ugly. Conceptually, if a virtual address is not enough to distinguish different computational resource, might be huge problems lying around. Isn't it contradict Turing model a process used to hold?

[jbrodman]

A program that tries to use VAs across contexts is an incorrect program. It is not expected or required to work.

On Mon, Nov 9, 2020 at 9:43 PM Zhong, Tsao notifications@github.com wrote:

I brought up this to attention because if a unique virtual address could point to at least two different context resource, when migration happens it'll be very ugly. Conceptually, if a virtual address is not enough to distinguish different computational resource, might be huge problems lying around. Isn't it contradict Turing model a process used to hold?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/intel/llvm/issues/2721#issuecomment-724413324, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOWWAWF7DSN6G2EBADZUXTSPCSDJANCNFSM4TIHB5MQ .

[CaoZhongZ]

Users could control that? All users to do is allocate address1 from context1, allocate address2 from context2, do works. But address1 and address2 could be the same number. right?

[jbrodman]

They could be, they could not. The point is, it's illegal, and writing programs on UB is a recipe for pain. If you need to share data between contexts, explicitly copy it.

On Mon, Nov 9, 2020 at 9:49 PM Zhong, Tsao notifications@github.com wrote:

Users could control that? All users to do is allocate address1 from context1, allocate address2 from context2, do works. But address1 and address2 could be the same number. right?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/intel/llvm/issues/2721#issuecomment-724415027, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOWWARX4R5MSF5UBQVD7NTSPCS2TANCNFSM4TIHB5MQ .

[keryell]

I have some issues understanding how a correct SYCL USM implementation could return the same C++ user-level address for 2 different objects. Even if there are various back-ends and contexts, at the end they have to cooperate with the OS for not fighting for the same addresses... At least something like Linux HMM can insure that even if behind the curtain you have actually different devices in different contexts and back-ends with different distributed physical memory pages, the illusion of a unique virtual address will hold. In the same way as if new int and another new int are returning the same value, you are in trouble (but it is still possible with out-of-memory condition, in the case of noexcept new).

[jbrodman]

For the shared/host allocations you're 100% right.

I think the only place where it's possible would be for device allocations which aren't accessible on the host.

Anything accessible on the host can't have this sort of conflict or reality would explode.

On Tue, Nov 10, 2020 at 2:26 PM Ronan Keryell notifications@github.com wrote:

I have some issues understanding how a correct SYCL USM implementation could return the same C++ user-level address for 2 different objects. Even if there are various back-ends and contexts, at the end they have to cooperate with the OS for not fighting for the same addresses... At least something like Linux HMM can insure that even if behind the curtain you have actually different devices in different contexts and back-ends with different distributed physical memory pages, the illusion of a unique virtual address will hold. In the same way as if new int and another new int are returning the same value, you are in trouble (but it is still possible with out-of-memory condition, in the case of noexcept new).

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/intel/llvm/issues/2721#issuecomment-724915901, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOWWASFKIASREYXBA6IUW3SPGHWZANCNFSM4TIHB5MQ .

[keryell]

While it would be possible, then it prevents use cases such as:

• you have a lot of pointers to some objects allocated in some various slabs on various devices by C++ allocators and you cannot std::partition/std::sort/... the pointer set according to the various address ranges so you can the process them on the right devices.

So, if the application/architecture/OS/... is not short on address spaces, it seems clearer to have unique virtual address space mapping, even for non host-accessible memory. Even if we are talking about pointers here, the audience is not the Linux kernel developers... :-)

[CaoZhongZ]

Agree, we have plenty space for a clear implementation and it is then much more convenient.

Other cases like, C++ programming language required to avoid single address, two object of same type scenario. Or programmer's heuristic on pointer arithmetic, different object would have different address, etc. You could feel things will break if we don't do it right.

[KornevNikita]

Hi! There have been no updates for at least the last 60 days, though the ticket has assignee(s).

@jbrodman, could I ask you to take one of the following actions? :)

• Please provide an update if you have any (or just a small comment if you don't have any yet).
• OR mark this issue with the 'confirmed' label if you have confirmed the problem/request and our team should work on it.
• OR close the issue if it has been resolved.
• OR take any other suitable action.

Thanks!

[github-actions[bot]]

Hi! There have been no updates for at least the last 60 days, though the issue has assignee(s).

@jbrodman, could you please take one of the following actions:

• provide an update if you have any
• unassign yourself if you're not looking / going to look into this issue
• mark this issue with the 'confirmed' label if you have confirmed the problem/request and our team should work on it
• close the issue if it has been resolved
• take any other suitable action.

Thanks!

[github-actions[bot]]

Hi! There have been no updates for at least the last 60 days, though the issue has assignee(s).

@jbrodman, could you please take one of the following actions:

• provide an update if you have any
• unassign yourself if you're not looking / going to look into this issue
• mark this issue with the 'confirmed' label if you have confirmed the problem/request and our team should work on it
• close the issue if it has been resolved
• take any other suitable action.

Thanks!