Error when using querySetting(nimcacheDir)

[arkanoid87]

I've found this problem while working with Futhark: https://github.com/PMunch/futhark/issues/12

Futhark generates a nim file in querySetting(nimcacheDir) at compile time, and then import it

It works, but vscode extensions error out due to nim check trying to import it from *_check cache folder, instead of [_d|_r]

I've created a minimal test to replicate the problem

vscodecache.nim

import std/[os, compilesettings, macros]

const cacheFile = querySetting(nimcacheDir) / "test.nim"

static:
  writeFile(cacheFile, "let foo* = 42")

macro loadFromCache: untyped =
  return quote do:
    include `cacheFile`

loadFromCache()
echo foo

nim c -r vscodecache.nim

42

vscode error == nim check vscodecache.nim

nim check vscodecache.nim
Hint: used config file '/home/jack/.choosenim/toolchains/nim-1.6.0/config/nim.cfg' [Conf]
Hint: used config file '/home/jack/.choosenim/toolchains/nim-1.6.0/config/config.nims' [Conf]
.......................................................................................
/home/jack/nim/vscodecache/vscodecache.nim(13, 14) template/generic instantiation of `loadFromCache` from here
/home/jack/nim/vscodecache/vscodecache.nim(9, 10) Error: cannot open file: /home/jack/.cache/nim/vscodecache_check/test.nim
/home/jack/nim/vscodecache/vscodecache.nim(14, 6) Error: undeclared identifier: 'foo'
candidates (edit distance, scope distance); see '--spellSuggest': 
 (2, 1): 'os' [module declared in /home/jack/nim/vscodecache/vscodecache.nim(1, 11)]
 (2, 3): 'bool' [type declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/basic_types.nim(15, 3)]
 (2, 3): 'io' [module declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system.nim(3132, 14)]
 (2, 3): 'low' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system.nim(1584, 6)]
 (2, 3): 'low' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system.nim(386, 6)]
 (2, 3): 'low' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system.nim(397, 6)]
 (2, 3): 'low' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system.nim(408, 6)]
 (2, 3): 'low' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system.nim(420, 6)]
 (2, 3): 'low' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system.nim(434, 6)]
 (2, 3): 'low' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system.nim(445, 6)]
 (2, 3): 'low' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system.nim(451, 6)]
 (2, 3): 'mod' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(216, 6)]
 (2, 3): 'mod' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(225, 6)]
 (2, 3): 'mod' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(226, 6)]
 (2, 3): 'mod' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(227, 6)]
 (2, 3): 'mod' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(228, 6)]
 (2, 3): 'mod' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(394, 6)]
 (2, 3): 'mod' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(397, 6)]
 (2, 3): 'mod' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(398, 6)]
 (2, 3): 'mod' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(399, 6)]
 (2, 3): 'mod' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(400, 6)]
 (2, 3): 'not' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system.nim(123, 6)]
 (2, 3): 'not' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(166, 6)]
 (2, 3): 'not' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(173, 6)]
 (2, 3): 'not' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(174, 6)]
 (2, 3): 'not' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(175, 6)]
 (2, 3): 'not' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(176, 6)]
 (2, 3): 'not' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(323, 6)]
 (2, 3): 'not' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(325, 6)]
 (2, 3): 'not' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(326, 6)]
 (2, 3): 'not' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(327, 6)]
 (2, 3): 'not' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(328, 6)]
 (2, 3): 'not' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/basic_types.nim(45, 6)]
 (2, 3): 'of' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system.nim(917, 6)]
 (2, 3): 'on' [const declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/basic_types.nim(19, 3)]
 (2, 3): 'or' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system.nim(117, 6)]
 (2, 3): 'or' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(302, 6)]
 (2, 3): 'or' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(307, 6)]
 (2, 3): 'or' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(308, 6)]
 (2, 3): 'or' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(309, 6)]
 (2, 3): 'or' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(310, 6)]
 (2, 3): 'or' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(351, 6)]
 (2, 3): 'or' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(353, 6)]
 (2, 3): 'or' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(354, 6)]
 (2, 3): 'or' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(355, 6)]
 (2, 3): 'or' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(356, 6)]
 (2, 3): 'or' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/basic_types.nim(53, 6)]
 (2, 3): 'or' [template declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/core/macros.nim(196, 10)]
 (2, 3): 'pop' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system.nim(1854, 6)]
 (2, 3): 'xor' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(312, 6)]
 (2, 3): 'xor' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(317, 6)]
 (2, 3): 'xor' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(318, 6)]
 (2, 3): 'xor' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(319, 6)]
 (2, 3): 'xor' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(320, 6)]
 (2, 3): 'xor' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(358, 6)]
 (2, 3): 'xor' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(360, 6)]
 (2, 3): 'xor' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(361, 6)]
 (2, 3): 'xor' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(362, 6)]
 (2, 3): 'xor' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/arithmetics.nim(363, 6)]
 (2, 3): 'xor' [proc declared in /home/jack/.choosenim/toolchains/nim-1.6.0/lib/system/basic_types.nim(58, 6)]

not sure if the fix should be in futhark or here, but it's worth discussing for libraries like Futhark that works by generating custom nim files in cache folder.

[saem]

Generating source in a cache just seems like a terrible idea. It's a cache, it can disappear with unknown lifetime.

The complexity of managing an index where live edits are happening to the source data is challenging enough, making that source data further transient or the transient nature of it ambiguous by having mixed use cache... I'm tired even thinking about it.

Unless someone (not it!) put in an incredible amount of work figuring out a reasonable data model and management of IO to support indexing transient data cleanly I wouldn't accept the complexity.

The error is either because:

1. nim check has a bad habit of subtlety deviating from the rest of the compiler and this is one such occasion.
2. nim check is not being passed the same set of flags etc as the compilation invocation and so it can't find the included source.

[arkanoid87]

The Futhark generated nim file is a cache. If not present in cache folder or if outdated is re-generated: https://github.com/PMunch/futhark/blob/875a7f5cf727483bc4c4d394f7d3c23699002aa8/src/futhark.nim#L466 It just exist to speed up things afaik.

Said that, I don't know enough details to say that using a cache folder this way is wrong, but I'd like to know if you think that a library should keep it's own cache for this purpose.

[PMunch]

As @arkanoid87 said, the file is a cache file. The macro itself can take quite a long time to generate code, something which would be annoying to have to do every time your rebuild. The cache file also allows editors to see this as a pure import and not time out, allowing them to work at all with Futhark (tested with NimLSP). If the file doesn't exist it is re-generated, or if the user changes anything about the import it also re-generates.

NimLSP doesn't seem to struggle with this, as long as you compile it once so the cache files are built then NimLSP will see them and allow you to follow definitions into the cache file, and get hover information etc.

Not entirely sure what nim check does differently, but it shouldn't be that hard to make it work. In the meantime I guess I could add a flag to Futhark which tells it to look in another folder than the cache folder for its cache files, so that VSCode users will be able to more easily use it.

[saem]

In the readme are instructions as to how to setup the extension dev environment, the slowest thing about it is likely waiting for downloads, but it's dead simple.

Anyhow, I'd open up a debug version of the extension with some console log statements for the nim check call. Just make sure all the flags you expect are there when nim check is called on the output.

A less convincing way to verify it is what's the command you use to compile your project? If it takes a bunch of flags and those aren't in the config then there not being given to nim check.

[arkanoid87]

@saem the code in first comment replicates the problem in vscode without any config.nims or command line option

just with nim c -r vscodecache.nim you get working runtime but error in vscode/nim check, and with nim check vscodecache.nim you get the error I see with vscode extension.

consider that <cache>/vscodecache_d folder must before running

[saem]

Implied paths work differently between nim compile and nim check, I found out because I ran into the issue a while back (can't recall more).