Logical vs Physical Pod Software Discussion

[dpetrisko]

Hi, we're running into issues when trying to index subpods that are not part of the main manycore array.

Our setup is (simplified):

                    [ vcache (0,1)     ]
[foo subpod (1, 0)] [ compute (1, 1) ]
                    [ vcache (2, 1)    ]

So to index foo subpod CSR 0x800, we use bsg_global_pod_ptr(px,py,x,y,local_addr) and do bsg_global_pod_ptr(0,1,15,0,0x800). However, this ended up giving an unexpected output address. Looking closer at the macro:

https://github.com/bespoke-silicon-group/bsg_manycore/blob/911ca3ab4c393cf89a320487edc95f2b3a979a0c/software/bsg_manycore_lib/bsg_manycore_arch.h

// pod remote pointer
// px = pod id x
// py = pod id y
// x = subcord x
// y = subcord y
#define bsg_global_pod_ptr(px,py,x,y,local_addr) \
  ((bsg_remote_int_ptr) ( (1 << GLOBAL_PREFIX_SHIFT)    \
                        | ((((1+(py*2))*bsg_global_Y)+y) << GLOBAL_Y_CORD_SHIFT)  \
                        | ((((px+1)*bsg_global_X)+x) << GLOBAL_X_CORD_SHIFT)  \
                        | ((int) local_addr)))

It appears that bsg_global_pod_ptr indexes some higher abstraction of px,py == the coordinate of compute in the [vcache, compute, vcache unit]. From offline discussion it seems like there is some nomenclature about "logical pod" vs "physical pod" that is missing from the reference doc. Can we clarify that? Secondly, we'll define a macro that just uses the pod coordinates passed directly as:

// pod remote pointer
// px = pod id x
// py = pod id y
// x = subcord x
// y = subcord y
#define bsg_global_physical_pod_ptr(px,py,x,y,local_addr) \
  ((bsg_remote_int_ptr) ( (1 << GLOBAL_PREFIX_SHIFT)    \
                        | ((((py)*bsg_global_Y)+y) << GLOBAL_Y_CORD_SHIFT)  \
                        | (((px*bsg_global_X)+x) << GLOBAL_X_CORD_SHIFT)  \
                        | ((int) local_addr)))

Whatever the correct nomenclature is, can we have both in the manycore library?

Minor, but I also suggest refactoring bsgio* macros to use the physical pod ptr as well. Currently they use hardcoded macros BSG_IO_X_CORD, and it'll be a bit more clear to use the physical pod ptr specifically?

[taylor-bsg]

@mrutt92 @tommydcjung @gaozihou

[mrutt92]

It appears that bsg_global_pod_ptr indexes some higher abstraction of px,py == the coordinate of compute in the [vcache, compute, vcache unit]. From offline discussion it seems like there is some nomenclature about "logical pod" vs "physical pod" that is missing from the reference doc. Can we clarify that? Secondly, we'll define a macro that just uses the pod coordinates passed directly as:

As someone privy to that offline discussion, this nomenclature was something i made up on the spot and was not widely adopted by bsg_manycore maintainers.

My understanding is that these macros were designed to address the local memory of vanilla-core tiles on other pods in the system. They were not designed necessarily with memory mapped IO registers or other types of endpoints in mind. Apparently the hardware supports it? Have you tried out your proposed macro?

[dpetrisko]

this nomenclature was something i made up on the spot

all the best nomenclature is

Apparently the hardware supports it?

Those BlackParrots hanging off of BigBlade need to do something after all :)

Have you tried out your proposed macro?

Yep, seems to work for our chip example

[mrutt92]

well then... what's there left to discuss?

[tommydcjung]

vcaches themselves aren't really a pod though... If you want to reach tiles in other pods using global EVA, you can use bsg_global_pod_ptr(). If you want to reach vcaches in other pods using global EVA (not sure why you would want to do that..), then you can set y as -1 or bsg_global_Y+1 in bsg_global_pod_ptr()?

[mrutt92]

I think the motivation is not to address the vcaches so much as other endpoint types on remote pods such as a blackparrot instance or a memory mapped fifo.

[dpetrisko]

vcaches themselves aren't really a pod though

I guess that's my high level question -- what's a "pod" in software? Is it the manycore compute array? Is it compute array+2*vcache? In hardware it seems clear since vcaches have their own pod coordinate, blackparrot has its own pod coordinate, etc.

[mrutt92]

i think having a macro that lets sw address all the memory mapped hw modules on the chip is valuable. i also think having another macro that abstracts away those modules and lets sw treat the chip like a grid of vanilla-core pods is also valuable. clearly, both interfaces have value depending on what the software is trying to do. one can be implemented in terms of the other.

[mrutt92]

depending on who you are, the word "pod" has a different meaning. it seems to me like it's an overloaded term and that could be the source of some confusion here.

[tommydcjung]

At the end of the day, it's just a macro doing some address calculation. You can add it here or just have it in your own code. Fine with me either way

[mrutt92]

well said