Diverging CRC16 results

[ClaesClaes]

HI, I'm experimenting with your library using CRC16 to compare if json files on an ESP32 and on an iPhone are the same. I have an iOS app that sends json files to an ESP32 which needs to stay in sync. I'm getting divergent CRC results between your lib and the lib I use in the iOS app.

The polynomial are the same (0x1021) in both libs and I'm setting the BigEndianIn (true or false) option to match the same option in the iOS lib. Any ideas what I could be doing wrong, or a tip on how I can troubleshoot this?

ESP32 code:

uint16_t calculateCRC(fs::FS &fs, const String& filename, const String& extension) {
    if (fs.exists("/" + filename + extension)) {
      File file = fs.open("/" + filename + extension);
      crc.reset();
      crc.setPolynome(0x1021);
      crc.setReverseIn(true); //Also tried (false) 
      while (file.available()) {
        uint8_t c = file.read();
        crc.add(c);  
      } 
     return crc.getCRC();
  }
}

Swift code: Usage:

func calculateCRC(data: Data) -> UInt16 {
        let byteArray: [UInt8] = Array(data)
        let crc16 = CRC16()
        crc16.append(byteArray)
        return crc16.currentValue
    }

Class:

/// Class to conveniently calculate CRC-16. It uses the CRC16-CCITT polynomial (0x1021)  by default
public class CRC16: CRC {

    /// The table that stores the precomputed remainders for all possible divisions for 1 byte
    /// This table is used as a lookup (to speed up the calculation)
    public let lookupTable: [UInt16]

    private(set) public var currentValue: UInt16 = 0

    /// Creates the instance for calculating CRC
    /// - Parameter polynomial: The polynomial to use. It uses CRC16-CCITT's polynomial (0x1021) by default
    public init(polynomial: UInt16 = 0x1021) {
        /// Generates the lookup table (make sure to generate it only once)
        self.lookupTable = (0...255).map { Self.crc16(for: UInt8($0), polynomial: polynomial) }
    }
    public func append(_ bytes: [UInt8]) {
        currentValue = crc16(for: bytes, initialValue: currentValue)
    }
    public func reset() {
        currentValue = 0
    }
}

extension CRC16 {
    /// Caculates CRC-16 of an array of Bytes (UInt8)
    private func crc16(for inputs: [UInt8], initialValue: UInt16 = 0) -> UInt16 {
        inputs.reduce(initialValue) { remainder, byte in
            let bigEndianInput = UInt16(byte).bigEndian
            let index = (bigEndianInput ^ remainder) >> 8
            return lookupTable[Int(index)] ^ (remainder << 8)
        }
    }
}

[RobTillaart]

Not much time this week to dive into this.

Things I would check

• line endians "\n \r" or single versions.
• UTF encoding?
• special characters?

Would start with a simple test - just a single char "a", extend it and see when it breaks.

[ClaesClaes]

No worries, thanks for the tip. Integers fitting in one byte produce the same crc results. So good first step. I have no special characters in the files (json) but could be something with the UTF encoding. I'll look in that direction. Thanks again.

[ClaesClaes]

Well, turns out both libraries spits out the exact same crc... It was my mistake. In the iOS application I add the crc number to the json file at the time of transmission to the ESP32 (my first verification architecture approach) but not the copy stored in iOS. So the two files could never be the same and obviously with different crc outcomes. Apologies for taking up your time.

[RobTillaart]

No problem, on contrary. It is good to have another real live test that the lib is working and doing what it supposed to do. Thanks