teragrep / pth_10

Data Processing Language (DPL) translator for Apache Spark
GNU Affero General Public License v3.0
0 stars 6 forks source link

Too long week/month/year relative modifier will create invalid timestamps #203

Closed StrongestNumber9 closed 6 months ago

StrongestNumber9 commented 7 months ago

Describe the bug

Too long relative modifier causes invalid timestamps

%dpl
index=x earliest=-999999999y

->

org.apache.spark.sql.streaming.StreamingQueryException: SQL [insert into <snip> ]; (conn=290793) Truncated incorrect datetime value: '169089589-04-28'

Expected behavior

Either limit/truncate the timestamp to SQL appropriate lengths or handle it as an error with good error description of what happened

How to reproduce

%dpl
index=x earliest=-999999999y

Screenshots

Software version

Desktop (please complete the following information if relevant):

Additional context

StrongestNumber9 commented 7 months ago

Note: Latest doesn't seem to be affected or has different threshold

eemhu commented 7 months ago

From the error message: ^\\d{4}\\/\\d{2}-\\d{2} seems like the maximum would be year 9999 and minimum year 1000. Perhaps need to check the year to be between these values.

eemhu commented 7 months ago

based on testing, 0 < epoch < int32 max value should be the valid range. However it might not be wise to lock the maximum value to that as it would only reach year 2038.

eemhu commented 7 months ago

internal pr merged