Some values from API have null byte/char, and therefore cannot be inserted to database directly

CybersecurityForDemocracy / tiktok-library

A library for interacting and large-scale data acquisition from the TikTok Research API

https://pypi.org/project/tiktok_research_api_helper/

4 stars 0 forks source link

Some values from API have null byte/char, and therefore cannot be inserted to database directly #91

Closed macpd closed 5 days ago

macpd commented 2 weeks ago

for example video ID 7420068284780875054 has 'username': '٠вяеииа٠ء\x00\x00\x00\x00\x00\x00\x00\x00'

I don't know if we should strip the null character from these strings (str.replace('\x00', '')), or unicode escape them (str.encode('unicode_escape'))