search

replikativ / datahike

A fast, immutable, distributed & compositional Datalog engine for everyone.
https://datahike.io
Eclipse Public License 1.0
1.63k stars 97 forks source link

Cannot open a relative filepath #76

Closed rschmukler closed 4 years ago

rschmukler commented 5 years ago

Hey! Thanks for this amazing library.

In my application I'd like to open a database at a relative file path. I've tried many permutations of paths but all seem to throw errors in one form or another. See below for my attempts and their results. Is this possible?

(d/create-database "datahike:file://./data/db")
1. Unhandled java.io.IOException
   Permission denied

(Presumably trying to write to /data instead of ./data)

(d/create-database "datahike:file:data/db")
;; or
(d/create-database "datahike:file:./data/db")

Both throw:

1. Unhandled java.lang.NullPointerException
   (No message)

             filestore.clj:  395  konserve.filestore/check-and-create-folder
             filestore.clj:  392  konserve.filestore/check-and-create-folder
             filestore.clj:  439  konserve.filestore/new-fs-store
             filestore.clj:  432  konserve.filestore/new-fs-store
               RestFn.java:  410  clojure.lang.RestFn/invoke
                store.cljc:   69  datahike.store$eval51949$fn__51951/invoke
              MultiFn.java:  229  clojure.lang.MultiFn/invoke
            connector.cljc:  147  datahike.connector$eval52150$fn__52157/invoke
            connector.cljc:   73  datahike.connector$eval52088$fn__52089$G__52075__52096/invoke
                  AFn.java:  156  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj:  660  clojure.core/apply
            connector.cljc:   85  datahike.connector$eval52140$fn__52143/doInvoke
               RestFn.java:  423  clojure.lang.RestFn/invoke
            connector.cljc:   73  datahike.connector$eval52088$fn__52089$G__52075__52096/invoke
            connector.cljc:  183  datahike.connector$create_database/invokeStatic
            connector.cljc:  182  datahike.connector$create_database/doInvoke
               RestFn.java:  410  clojure.lang.RestFn/invoke
                      REPL:  101  tabrasa.store/eval52344
                      REPL:  101  tabrasa.store/eval52344
             Compiler.java: 7177  clojure.lang.Compiler/eval
             Compiler.java: 7132  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj:  660  clojure.core/apply
                regrow.clj:   18  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   79  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   55  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  142  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  171  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  170  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  835  java.lang.Thread/run

(d/create-database "datahike:file://data/db")
1. Unhandled java.io.IOException
   No such file or directory

       UnixFileSystem.java:   -2  java.io.UnixFileSystem/createFileExclusively
                 File.java: 1024  java.io.File/createNewFile
             filestore.clj:  398  konserve.filestore/check-and-create-folder
             filestore.clj:  392  konserve.filestore/check-and-create-folder
             filestore.clj:  439  konserve.filestore/new-fs-store
             filestore.clj:  432  konserve.filestore/new-fs-store
               RestFn.java:  410  clojure.lang.RestFn/invoke
                store.cljc:   69  datahike.store$eval51949$fn__51951/invoke
              MultiFn.java:  229  clojure.lang.MultiFn/invoke
            connector.cljc:  147  datahike.connector$eval52150$fn__52157/invoke
            connector.cljc:   73  datahike.connector$eval52088$fn__52089$G__52075__52096/invoke
                  AFn.java:  156  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj:  660  clojure.core/apply
            connector.cljc:   85  datahike.connector$eval52140$fn__52143/doInvoke
               RestFn.java:  423  clojure.lang.RestFn/invoke
            connector.cljc:   73  datahike.connector$eval52088$fn__52089$G__52075__52096/invoke
            connector.cljc:  183  datahike.connector$create_database/invokeStatic
            connector.cljc:  182  datahike.connector$create_database/doInvoke
               RestFn.java:  410  clojure.lang.RestFn/invoke
                      REPL:  101  tabrasa.store/eval52350
                      REPL:  101  tabrasa.store/eval52350
             Compiler.java: 7177  clojure.lang.Compiler/eval
             Compiler.java: 7132  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj:  660  clojure.core/apply
                regrow.clj:   18  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   79  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   55  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  142  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  171  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  170  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  835  java.lang.Thread/run
kordano commented 5 years ago

Thanks, I'll add it as a test case and see how it could be fixed.

whilo commented 5 years ago

One thing to keep in mind that relative paths for this can be tricky because the JVM has its own classpath to which it calculates relative positions. If you use this in configuration files or scripts it might break easily. @rschmukler do you happen to know how other JVM storage solutions handle this?

rschmukler commented 5 years ago

@whilo I'm not too sure of the intricacies of getting the current working directory from the JVM. Historically I have used (.getAbsolutePath (io/file "")) to retrieve it. This works well, at least for Linux / MacOS based systems that I've used.

Looking at Raynes/fs it looks like they do something similar in their cwd implementation.

whilo commented 5 years ago

Yes, I think we can support it with this semantics in file URLs.

TimoKramer commented 4 years ago

should i try my luck @kordano

TimoKramer commented 4 years ago

While trying some solutions it seems to me that this is working as URIs are expected to be working. RFC3986 3.3 states following:

The path segments "." and "..", also known as dot-segments, are defined for relative reference within the path name hierarchy. They are intended for use at the beginning of a relative-path reference (Section 4.2) to indicate relative position within the hierarchical tree of names. This is similar to their role within some operating systems' file directory structures to indicate the current directory and parent directory, respectively. However, unlike in a file system, these dot-segments are only interpreted within the URI path hierarchy and are removed as part of the resolution process (Section 5.2). This means when using a URI for configuration it is not supposed to work with relative file paths like one would expect coming from a CLI. Relative file paths are only solved relative to the first path segement like /tmp/ or defaults to root /. Some examples:

There is no such thing like $PWD or $HOME for a URI. You can even try this in your browser.

A solution would be to hack around with the string returned from the URI-object. I would not recommend that. Better is to use a map as for configuring datahike like it is already possible and even more so with the new configuration approach.

This issue should be resolved as invalid.

rschmukler commented 4 years ago

@TimoKramer thanks for taking the time to research this and provide the explaination. I just confirmed that the relative path in the {:backend :file :path _} did indeed work. Sorry for the hassle!