aespinilla / Tiktoken

Openai's tiktoken implementation written in Swift
MIT License
49 stars 12 forks source link

Fatal Error due to Force-Unwrapping in `CoreBPE.bytePairEncode(piece: ranks:)` #5

Open gdbing opened 5 months ago

gdbing commented 5 months ago

Load.fetch(stringURL:) method invokes try await URLSession.shared.data(from: url) to fetch encoder the first time it is used. If an error occurs (e.g., the method is invoked without networking permissions), it throws an exception.

In the calls to the fetch method, this exception is handled with try?, substituting an empty collection [:] when an error is caught.

In the CoreBPE.bytePairEncode(piece: ranks:) method, there's a force-unwrapping operation return [ranks[piece]!], which causes a fatal error if ranks is an empty collection.

Recommendation: Catching an error from Load.fetch(stringURL:) should fail more decisively, halting execution, instead of progressing with an empty encoder.