fix bug during w2v training with utf8 characters

kakao / buffalo

TOROS Buffalo: A fast and scalable production-ready open source project for recommender systems

Apache License 2.0

576 stars 106 forks source link

fix bug during w2v training with utf8 characters #75

Closed hugh-ga closed 11 months ago

hugh-ga commented 11 months ago

bug

when training w2v with Korean words(utf-8 characters), idmap['cols'] couldn't get utf-8

UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

as-is

dtype=Sx
- S: data type should be ascii characters

to be

dtype=h5py.string_dtype('utf-8')
- docs

CLAassistant commented 11 months ago

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}