Closed bendichter closed 2 years ago
@bendichter The code that adds the --dandiset-path
option is currently commented out for some reason, so you have no choice but to be in the Dandiset directory or a subdirectory thereof.
@yarikoptic Should the --dandiset-path
option be uncommented, or should the references to it be removed from the documentation?
@jwodder thanks for the quick response! I would very much like to be able to upload a directory that is not the cwd
@bendichter What exactly is preventing you from cd
ing to that directory before running dandi upload
?
If you really need to upload a non-PWD directory, you'll have to call dandi.upload.upload()
yourself; cf. how dandi upload
does it.
@jwodder
I am confused by the upload
function as well. It has a required arg of paths
and an optional arg of dandiset_path
, and no docstring to explain the difference between these args. There is also no type hint to indicate what type "paths" should be. A list of strings? Should I pass in an empty list and use the dandiset_path
optional arg? Could you please add a docstring so I know how to use this function?
@bendichter Type hints were added in #887; the function looks like this now:
I'm not going to write out a full docstring right now, but for the arguments in question:
dandiset_path
needs to be the directory containing the uploaded Dandiset's dandiset.yaml
or a subdirectory thereof; if not specified, it defaults to the current directory.paths
is a list of files & directories inside dandiset_path
to upload; if it's empty, all assets in the Dandiset are uploaded.@jwodder ok thanks that's what I needed to know
@jwodder how would you feel about making paths
optional?
@bendichter Done: https://github.com/dandi/dandi-cli/pull/899
@bendichter What exactly is preventing you from
cd
ing to that directory before runningdandi upload
?
and wouldn't dandi upload the-path-to-dandiset-top-directory/
work? --dandiset-path
is really to just establish where the "top" of the dandiset is, may be too inspired by datalad's UI.
edit: i.e. should we do anything extra for this issue or just consider it resolved?
@yarikoptic That wouldn't work, as upload
would still start looking in the current directory for the dandiset.yaml
file. If an explicit path given on the command line isn't under that, you'll get an error.
should we do anything extra for this issue or just consider it resolved?
We need to decide whether to uncomment --dandset-path
or remove its references from the documentation.
Dandiset.find
(which is ran on dandiset_path
) already uses find_parent_directory_containing
and thus possibly ascending path hierarchy, so dandiset_path
can point to some other location within dandiset and thus being not really "strict" for what --dandiset-path
points to.
So, overall I am still not sure on whether we should keep that dandiset_path
at all since it doesn't really serve any pragmatic purpose (unlike in datalad where it might define the "scope" of operation in particular across dataset boundaries), and instead just make handling of paths
a bit smarter.
paths
not specified -- take cwd
(so -- current behavior)paths
is specified
/
or empty string on Windows having different drives ;-))Dandiset.find()
on that common prefix. If nothing found -- just inform user on what was that common path in the error to be issued.WDYT @jwodder ?
@yarikoptic That sounds like a good strategy, but I think we should keep the Dandiset-finding behavior the same between dandi upload
and dandi organize
(the only command that currently has a --dandiset-path
option). For the record, its current behavior is: if --dandiset-path
is specified, use that (without recursing upwards); otherwise, recurse upwards from the current directory in search of dandiset.yaml
. What strategy should be preferred for both commands?
re behavior if command has a --dandiset-path
argument -- totally agree, they should behave identically. organize
seems to use Dandiset.find
from CWD only no dataset-path is specified, and if specified -- takes it as is, not search is performed. I think such behavior is ok and explicit.
organize
does need --dandiset-path
as the one to provide location where to organize into if cannot be deduced from CWD.
upload
-- I feel it does not need --dandiset-path
since if paths
are specified -- they pretty much instruct about the location of dandiset-path
(hence above comment), and if not specified -- deduce from CWD to upload entire dandiset user is under. Then we could just remove handling of dandiset_path
from upload and not bother harmonizing its behavior to organize
. Agree?
@yarikoptic Agreed: https://github.com/dandi/dandi-cli/pull/916
dandi upload --help
mentions a--dandiset-path
arg, but I don't see one in the options. What is the proper way to set this without being in that directory?