ocaml-community / sedlex

An OCaml lexer generator for Unicode
MIT License
235 stars 43 forks source link

Position information of buffers #103

Closed chengtie closed 3 years ago

chengtie commented 3 years ago

I'm translating a lexer in ocamllex to sedlex.

In ocamllex, the following code prints well token positions:

(* in e.ml *)
Parser_e.main (Lexer_e.debug Lexer_e.token_ao) lexbuf
(* in lexer_e.mll *)
let debug rule = fun lexbuf ->
  let result = rule lexbuf in
  print_endline (string_of_token result);
  let pos = lexbuf.lex_curr_p in
  pffo "%s:%d:%d\n" pos.pos_fname pos.pos_lnum (pos.pos_cnum - pos.pos_bol + 1);
  result

With sedlex, the same function for debug raises an error Unbound record field lex_curr_p at let pos = lexbuf.lex_curr_p in.

(* in e.ml *)
let lexbuf = Sedlexing.Utf8.from_string s in
let lexer = Sedlexing.with_tokenizer (Sedlexer_e.debug Sedlexer_e.token_rc) lexbuf in
let parser = MenhirLib.Convert.Simplified.traditional2revised Parser_e.main in
parser lexer

Does anyone know how I could get the position information of buffers?

chengtie commented 3 years ago

I found it, it is lexing_positions.