ERROR: Ran out of tokens

[ParkerrDev]

On any example HolyC program I've tried, when I go to compile it I will get an:

ERROR: line 7: Ran out of tokens

on an obscure line number.

The program:

U0 Function(U8 *name="Bob")
{
  "%s\n",name;
}

Function; 
Function("Terry");   

Any idea why this happens? Am I doing something wrong? How can I fix it?

[Jamesbarford]

Hi, thanks for this. What OS are you on? I've tried the above code on an intel mac and it works fine.

The global function calling mechanism is not as robust as having a main function, oftentimes wrapping your code in a Main can solve some issues:

U0 Function(U8 *name="Bob")
{
  "%s\n",name;
}

U0 Main()
{
  Function; 
  Function("Terry");
}

[ParkerrDev]

I have tried it on both NixOS and a Fedora Linux Server, both had the same error. Both use intel processors. To debug it, maybe spin up a live Linux server somewhere?

[Jamesbarford]

I have now tried this on a couple different machines (Linux) and am not able to re-create this error.

I was going to ask what version of the compiler you are using, however I've not implemented versioning 😂. So all I can ask is are you using the code from the branch main?

[ParkerrDev]

Yes I cloned main.

Here is HolySpirit.HC (same as your code above)

U0 Function(U8 *name="Bob")
{
  "%s\n",name;
}

U0 Main()
{
  Function; 
  Function("Terry");
}

Here are the exact commands I used before I arrived at the issue:

[parker@jarvis ~]$ git clone https://github.com/Jamesbarford/holyc-lang.git
Cloning into 'holyc-lang'...
remote: Enumerating objects: 837, done.
remote: Counting objects: 100% (470/470), done.
remote: Compressing objects: 100% (173/173), done.
remote: Total 837 (delta 369), reused 323 (delta 297), pack-reused 367 (from 1)
Receiving objects: 100% (837/837), 711.86 KiB | 2.89 MiB/s, done.
Resolving deltas: 100% (592/592), done.
[parker@jarvis ~]$ cd holyc-lang

make

[parker@jarvis holyc-lang]$ make
cmake -S ./src -B ./build -G 'Unix Makefiles' \
        -DCMAKE_C_COMPILER=gcc \
        -DCMAKE_BUILD_TYPE=Release \
        && make -C ./build -j2
-- The C compiler identification is GNU 13.3.1
-- The CXX compiler identification is GNU 13.3.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (1.7s)
-- Generating done (0.0s)
-- Build files have been written to: /home/parker/holyc-lang/build
make[1]: Entering directory '/home/parker/holyc-lang/build'
[ 11%] Building C object CMakeFiles/hcc.dir/aostr.c.o
[ 11%] Building C object CMakeFiles/hcc.dir/ast.c.o
[ 16%] Building C object CMakeFiles/hcc.dir/cctrl.c.o
[ 22%] Building C object CMakeFiles/hcc.dir/cfg-print.c.o
[ 27%] Building C object CMakeFiles/hcc.dir/cfg.c.o
[ 33%] Building C object CMakeFiles/hcc.dir/compile.c.o
[ 38%] Building C object CMakeFiles/hcc.dir/dict.c.o
[ 44%] Building C object CMakeFiles/hcc.dir/lexer.c.o
[ 50%] Building C object CMakeFiles/hcc.dir/list.c.o
[ 55%] Building C object CMakeFiles/hcc.dir/main.c.o
[ 61%] Building C object CMakeFiles/hcc.dir/map.c.o
[ 66%] Building C object CMakeFiles/hcc.dir/memory.c.o
[ 72%] Building C object CMakeFiles/hcc.dir/parser.c.o
[ 77%] Building C object CMakeFiles/hcc.dir/prsasm.c.o
[ 83%] Building C object CMakeFiles/hcc.dir/prslib.c.o
[ 88%] Building C object CMakeFiles/hcc.dir/prsutil.c.o
[ 94%] Building C object CMakeFiles/hcc.dir/x86.c.o
[100%] Linking C executable /home/parker/holyc-lang/hcc
[100%] Built target hcc
make[1]: Leaving directory '/home/parker/holyc-lang/build'

sudo make install

[parker@jarvis holyc-lang]$ sudo make install
[sudo] password for parker: 
make -C ./build install
make[1]: Entering directory '/home/parker/holyc-lang/build'
[100%] Built target hcc
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/bin/hcc
-- Installing: /usr/local/include/tos.HH
gcc -fPIC -c /tmp/holyc-asm.s -o ./all.o
ar rcs tos.a all.o
cp -pPR ./tos.a /usr/local/lib/libtos.a
make[1]: Leaving directory '/home/parker/holyc-lang/build'
[parker@jarvis holyc-lang]$ which hcc
/usr/local/bin/hcc

As you can see here, I am in my home directory trying multiple ways to compile HolySpirit.HC, and it fails with the token error:

[parker@jarvis ~]$ hcc -run ./HolySpirit.HC
ERROR: line 9: Ran out of tokens
[parker@jarvis ~]$ hcc ./HolySpirit.HC
ERROR: line 9: Ran out of tokens
[parker@jarvis ~]$ hcc -run HolySpirit.HC
ERROR: line 9: Ran out of tokens
[parker@jarvis ~]$ ./holyc-lang/hcc ./HolySpirit.HC
ERROR: line 9: Ran out of tokens
[parker@jarvis ~]$ 

Heres a fastfetch from the Fedora Server:

[parker@jarvis ~]$ fastfetch --logo none
-------------
OS: Fedora Linux 39 (Server Edition) x86_64
Host: 90HV0003US (ideacentre 510A-15ICB)
Kernel: Linux 6.10.6-100.fc39.x86_64
Uptime: 1 day, 19 hours, 54 mins
Packages: 1195 (rpm)
Shell: bash 5.2.26
Cursor: Adwaita
Terminal: node
CPU: Intel(R) Core(TM) i5-8400 (6) @ 4.00 GHz
GPU: Intel UHD Graphics 630 @ 1.05 GHz [Integrated]
Memory: 1.79 GiB / 7.62 GiB (23%)
Swap: 0 B / 7.62 GiB (0%)
Disk (/): 47.54 GiB / 114.94 GiB (41%) - xfs
Locale: en_US.UTF-8

[ParkerrDev]

Another issue worth noting:

When running the unit tests:

It initially didn't work and I had to install sqlite-devel, then the initial tests started to pass but it gets hung at 'Test - Simple Threading'

[parker@jarvis holyc-lang]$ make unit-test
make -C ./build unit-test
make[1]: Entering directory '/home/parker/holyc-lang/build'
[ 94%] Built target hcc
[100%] Building and running unit tests
Running unit tests...
Testing if else
PASSED: 4/4
=====
Test Assigning pointer to a pointer from a class, with brackets: PASSED: 1/1
====
Test Assigning pointer to a pointer from a class, no brackets: PASSED: 1/1
====
Test - Juggling an array MAlloc(I64 * 10): PASSED: 4/4
====
Test - Pointer to a pointer from a class: PASSED: 1/1
=====
Test - Taking the address of a local: PASSED: 1/1
Test - class definition: PASSED: 6/6
====
Test - Linked list recusive definition class: PASSED: 5/5
====
Test - 'break' and 'continue' inside while and for loops: PASSED: 4/4
====
Test - Default function parameters: PASSED: 1/1
====
Test - qsort, recursion and default function params: PASSED: 1/1
====
Assignment tests:
= test: PASSED: 2/2
+= test: PASSED: 2/2
-= test: PASSED: 2/2
*= test: PASSED: 2/2
/= test: PASSED: 2/2
%= test: PASSED: 2/2
&= test: PASSED: 2/2
|= test: PASSED: 2/2
^= test: PASSED: 2/2
>>= test: PASSED: 2/2
<<= test: PASSED: 2/2
=====
Test - Function pointers: PASSED: 5/5
====
Test - Simple Threading: 

After hittitng CTRL-C it skips the test and the rest seem to pass:

But it would hang there forever if I didn't. I wonder if this is a related to the token error problem. 🤔

^CFAILED to run: 13_threads.HC
Test Mathematics:
Subtraction - : PASSED: 7/7
Multiplication * : PASSED: 5/5
Modulo % : PASSED: 5/5
Addition + : PASSED: 5/5
Division / : PASSED: 3/3
MATHS_SHL << : PASSED: 3/3
MATHS_SHR >> : PASSED: 3/3
Xor ^: PASSED: 3/3
Or |: PASSED: 3/3
And &: PASSED: 3/3
Not ~: PASSED: 3/3
=====
Test - MemCpy, MemSet: PASSED: 4/4
Test - comma separated declarators & types: PASSED: 3/3
====
Test - Variable arguments: PASSED: 15/15
=====
Test - Type casting:
PASSED: 9/9
=====
Test - Switch statement: PASSED: 7/7
Loop test suite
Testing For Loop: PASSED: 2/2
Testing while loop: PASSED: 1/1
Testing do-while loop: PASSED: 1/1
=====
Testing GOTO: PASSED: 2/2
=====
Addressing: PASSED: 2/2
====
Malloced Arrays:
PASSED: 10/10
PASSED: 2/2
=====
Test: Union and class inheritance: PASSED: 4/4
====
Test Scripting: Test Scripting: PASSED: 4/4
===
Test - Floating point arithmetic:
PASSED: 5/5
====
Test - auto keyword: PASSED: 2/2
====
Test - Arrays & multi dimensional: PASSED: 4/4
====
Test - Preprocessor: PASSED: 6/6
====
Test - FzF fuzzy finding: PASSED: 7/7
====
Test - JSON parsing: {"hello": 10,"key": "some string","id": null,"bool2": false,"bool": true,"array": [1,30030303,430,-14]}
PASSED: 10/10
====
Test - sqlite bindings & abstraction: 
PASSED: 4/4
====
Test - local static variables and global variables: 
PASSED: 3/3
====
Test - Global and static array
PASSED: 2/2
====
Test - basic io: PASSED: 4/4
====
make[3]: *** [CMakeFiles/Makefile2:110: CMakeFiles/unit-test.dir/all] Interrupt
make[2]: *** [CMakeFiles/Makefile2:117: CMakeFiles/unit-test.dir/rule] Interrupt
make[1]: *** [Makefile:182: unit-test] Interrupt
make: *** [Makefile:18: unit-test] Interrupt