This code never used the loop variable in these two for loops so they are merely repeatedly using at the same sample in the FIFO multiple times instead of considering each sample as they should be. I believe you wanted to add i*4 to all of your buf indices or rework your for loop to something similar to for(int i=0; i<bytesRead; i+=4) and just use i as the buf index base.
https://github.com/adafruit/Adafruit_APDS9960/blob/master/Adafruit_APDS9960.cpp#L336
This code never used the loop variable in these two for loops so they are merely repeatedly using at the same sample in the FIFO multiple times instead of considering each sample as they should be. I believe you wanted to add i*4 to all of your buf indices or rework your for loop to something similar to for(int i=0; i<bytesRead; i+=4) and just use i as the buf index base.