Must be LITTLE/BIG Endian Host

[Terramoto]

I'm getting the following error when attempting to build it on a rpi4 with kernel 5.15.61.

#rm -f .symvers.8852au
make ARCH=arm64 CROSS_COMPILE= -C /lib/modules/5.15.61-v8+/build/ M=/home/terramot                                                                                                                 o/rtl8852au  modules
make[1]: Entering directory '/home/terramoto/linux-5b775d7293eb75d6dfc9c5ffcb95c50                                                                                                                 12cd0c3f8'
  CC [M]  /home/terramoto/rtl8852au/os_dep/osdep_service.o
In file included from /home/terramoto/rtl8852au/include/drv_types.h:26,
                 from /home/terramoto/rtl8852au/os_dep/osdep_service.c:17:
/home/terramoto/rtl8852au/include/rtw_byteorder.h:30:5: error: #error "Must be LIT                                                                                                                 TLE/BIG Endian Host"
   30 |  #  error "Must be LITTLE/BIG Endian Host"
      |     ^~~~~
make[2]: *** [scripts/Makefile.build:289: /home/terramoto/rtl8852au/os_dep/osdep_s                                                                                                                 ervice.o] Error 1
make[1]: *** [Makefile:1898: /home/terramoto/rtl8852au] Error 2
make[1]: Leaving directory '/home/terramoto/linux-5b775d7293eb75d6dfc9c5ffcb95c501                                                                                                                 2cd0c3f8'
make: *** [Makefile:639: modules] Error 2

$ uname -a
Linux RPI4 5.15.61-v8+ #1579 SMP PREEMPT Fri Aug 26 11:16:44 BST 2022 aarch64 GNU/Linux

Most likely something i'm doing wrong, just not sure what.

[lwfinger]

The code was dependent on the definition of CONFIG_LITTLE_ENDIAN. I changed that to use the kernel definition of __LITTLE_ENDIAN. Do a git pull and try again.

[Terramoto]

Many thanks, that got me through the initial error but i'm now getting the following:

                 from /home/terramoto/rtl8852au/core/rtw_security.c:17:
/home/terramoto/rtl8852au/core/rtw_security.c: In function ‘rtw_tkip_encrypt’:
/home/terramoto/rtl8852au/include/rtw_security.h:337:12: error: ‘union pn48’ has no member named ‘_byte_’
  337 |   dot11txpn._byte_.TSC0 = iv[2];\
      |            ^
/home/terramoto/rtl8852au/core/rtw_security.c:744:3: note: in expansion of macro ‘GET_TKIP_PN’
  744 |   GET_TKIP_PN(iv, dot11txpn);
      |   ^~~~~~~~~~~
/home/terramoto/rtl8852au/include/rtw_security.h:338:12: error: ‘union pn48’ has no member named ‘_byte_’
  338 |   dot11txpn._byte_.TSC1 = iv[0];\
      |            ^
/home/terramoto/rtl8852au/core/rtw_security.c:744:3: note: in expansion of macro ‘GET_TKIP_PN’
  744 |   GET_TKIP_PN(iv, dot11txpn);
      |   ^~~~~~~~~~~
/home/terramoto/rtl8852au/include/rtw_security.h:339:12: error: ‘union pn48’ has no member named ‘_byte_’
  339 |   dot11txpn._byte_.TSC2 = iv[4];\
      |            ^
/home/terramoto/rtl8852au/core/rtw_security.c:744:3: note: in expansion of macro ‘GET_TKIP_PN’
  744 |   GET_TKIP_PN(iv, dot11txpn);
      |   ^~~~~~~~~~~
/home/terramoto/rtl8852au/include/rtw_security.h:340:12: error: ‘union pn48’ has no member named ‘_byte_’
  340 |   dot11txpn._byte_.TSC3 = iv[5];\
      |            ^
/home/terramoto/rtl8852au/core/rtw_security.c:744:3: note: in expansion of macro ‘GET_TKIP_PN’
  744 |   GET_TKIP_PN(iv, dot11txpn);
      |   ^~~~~~~~~~~
/home/terramoto/rtl8852au/include/rtw_security.h:341:12: error: ‘union pn48’ has no member named ‘_byte_’
  341 |   dot11txpn._byte_.TSC4 = iv[6];\
      |            ^
/home/terramoto/rtl8852au/core/rtw_security.c:744:3: note: in expansion of macro ‘GET_TKIP_PN’
  744 |   GET_TKIP_PN(iv, dot11txpn);
      |   ^~~~~~~~~~~
/home/terramoto/rtl8852au/include/rtw_security.h:342:12: error: ‘union pn48’ has no member named ‘_byte_’
  342 |   dot11txpn._byte_.TSC5 = iv[7];\
      |            ^
/home/terramoto/rtl8852au/core/rtw_security.c:744:3: note: in expansion of macro ‘GET_TKIP_PN’
  744 |   GET_TKIP_PN(iv, dot11txpn);
      |   ^~~~~~~~~~~
/home/terramoto/rtl8852au/core/rtw_security.c: In function ‘rtw_tkip_decrypt’:
/home/terramoto/rtl8852au/include/rtw_security.h:337:12: error: ‘union pn48’ has no member named ‘_byte_’
  337 |   dot11txpn._byte_.TSC0 = iv[2];\
      |            ^
/home/terramoto/rtl8852au/core/rtw_security.c:842:4: note: in expansion of macro ‘GET_TKIP_PN’
  842 |    GET_TKIP_PN(iv, dot11txpn);
      |    ^~~~~~~~~~~
/home/terramoto/rtl8852au/include/rtw_security.h:338:12: error: ‘union pn48’ has no member named ‘_byte_’
  338 |   dot11txpn._byte_.TSC1 = iv[0];\
      |            ^
/home/terramoto/rtl8852au/core/rtw_security.c:842:4: note: in expansion of macro ‘GET_TKIP_PN’
  842 |    GET_TKIP_PN(iv, dot11txpn);
      |    ^~~~~~~~~~~
/home/terramoto/rtl8852au/include/rtw_security.h:339:12: error: ‘union pn48’ has no member named ‘_byte_’
  339 |   dot11txpn._byte_.TSC2 = iv[4];\
      |            ^
/home/terramoto/rtl8852au/core/rtw_security.c:842:4: note: in expansion of macro ‘GET_TKIP_PN’
  842 |    GET_TKIP_PN(iv, dot11txpn);
      |    ^~~~~~~~~~~
/home/terramoto/rtl8852au/include/rtw_security.h:340:12: error: ‘union pn48’ has no member named ‘_byte_’
  340 |   dot11txpn._byte_.TSC3 = iv[5];\
      |            ^
/home/terramoto/rtl8852au/core/rtw_security.c:842:4: note: in expansion of macro ‘GET_TKIP_PN’
  842 |    GET_TKIP_PN(iv, dot11txpn);
      |    ^~~~~~~~~~~
/home/terramoto/rtl8852au/include/rtw_security.h:341:12: error: ‘union pn48’ has no member named ‘_byte_’
  341 |   dot11txpn._byte_.TSC4 = iv[6];\
      |            ^
/home/terramoto/rtl8852au/core/rtw_security.c:842:4: note: in expansion of macro ‘GET_TKIP_PN’
  842 |    GET_TKIP_PN(iv, dot11txpn);
      |    ^~~~~~~~~~~
/home/terramoto/rtl8852au/include/rtw_security.h:342:12: error: ‘union pn48’ has no member named ‘_byte_’
  342 |   dot11txpn._byte_.TSC5 = iv[7];\
      |            ^
/home/terramoto/rtl8852au/core/rtw_security.c:842:4: note: in expansion of macro ‘GET_TKIP_PN’
  842 |    GET_TKIP_PN(iv, dot11txpn);
      |    ^~~~~~~~~~~
make[2]: *** [scripts/Makefile.build:289: /home/terramoto/rtl8852au/core/rtw_security.o] Error 1
make[1]: *** [Makefile:1898: /home/terramoto/rtl8852au] Error 2
make[1]: Leaving directory '/home/terramoto/linux-5b775d7293eb75d6dfc9c5ffcb95c5012cd0c3f8'
make: *** [Makefile:639: modules] Error 2

[parheliamm]

@lwfinger : After "rtw8852au: Remove reliance on old form of ENDIAN." commit: Build break on arm64 cross-compile as below:

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -C /home/lge/rk3588_repo_sdk_v1.0.2a/kernel M=/home/lge/rk3588_repo_sdk_v1.0.2a/kernel/drivers/staging/rtl8852au  modules
...
In file included from /home/lge/rk3588_repo_sdk_v1.0.2a/kernel/drivers/staging/rtl8852au/phl/hal_g6/hal_api_bb.c:17:
In file included from /home/lge/rk3588_repo_sdk_v1.0.2a/kernel/drivers/staging/rtl8852au/phl/hal_g6/phy/bb/halbb_export_fun.h:43:
/home/lge/rk3588_repo_sdk_v1.0.2a/kernel/drivers/staging/rtl8852au/phl/hal_g6/phy/bb/halbb_ch_info_ex.h:66:9: error: macro name must be an identifier
        #ifdef (__LITTLE_ENDIAN)
               ^
/home/lge/rk3588_repo_sdk_v1.0.2a/kernel/drivers/staging/rtl8852au/phl/hal_g6/phy/bb/halbb_ch_info_ex.h:75:9: error: macro name must be an identifier
        #ifdef (__LITTLE_ENDIAN)
               ^
/home/lge/rk3588_repo_sdk_v1.0.2a/kernel/drivers/staging/rtl8852au/phl/hal_g6/phy/bb/halbb_ch_info_ex.h:95:9: error: macro name must be an identifier
        #ifdef (__LITTLE_ENDIAN)
               ^
3 errors generated.

I suggest revert this change, to support arm64 other other platform, we could leverage this kind of codes in Makefile:

ifeq ($(CONFIG_PLATFORM_I386_PC), y)
EXTRA_CFLAGS += -DCONFIG_LITTLE_ENDIAN
EXTRA_CFLAGS += -DCONFIG_IOCTL_CFG80211 -DRTW_USE_CFG80211_STA_EVENT
EXTRA_CFLAGS += -DCONFIG_RADIO_WORK
#EXTRA_CFLAGS += -DCONFIG_CONCURRENT_MODE
SUBARCH := $(shell uname -m | sed -e s/i.86/i386/)
ARCH ?= $(SUBARCH)
CROSS_COMPILE ?=
KVER  ?= $(shell uname -r)
KSRC := /lib/modules/$(KVER)/build
MODDESTDIR ?= /lib/modules/$(KVER)/kernel/drivers/net/wireless/
INSTALL_PREFIX :=
STAGINGMODDIR := /lib/modules/$(KVER)/kernel/drivers/staging
ifeq ($(CONFIG_PCI_HCI), y)
EXTRA_CFLAGS += -DCONFIG_PLATFORM_OPS
_PLATFORM_FILES := platform/platform_linux_pc_pci.o
OBJS += $(_PLATFORM_FILES)
endif
endif

[Terramoto]

https://github.com/lwfinger/rtl8852au/issues/19#issuecomment-1295135825 this allowed me to compile it.

[parheliamm]

#19 (comment) this allowed me to compile it.

I revert that commit on my local branch, revert that patch is easy and safe.