search

datalad / datalad

Keep code, data, containers under control with git and git-annex
http://datalad.org
Other
525 stars 110 forks source link

python api create: cfg_proc results in exception #5018

Closed SteffenBrinckmann closed 3 years ago

SteffenBrinckmann commented 3 years ago

What is the problem?

If I run in python datalad.api.create(path='mydataset', cfg_proc='text2git') I get an exception UnsupportedOperation: fileno

What steps will reproduce the problem?

import datalad datalad.api.create(path='mydataset', cfg_proc='text2git')

What version of DataLad are you using (run datalad --version)? On what operating system (consider running datalad wtf)?

datalad 0.13.4

WTF

configuration <SENSITIVE, report disabled by configuration>

datalad

Is there anything else that would be useful to know in this context?

The exception log: [INFO] Creating a new annex repo at /home/sbrinckm/temporary_test0/mydataset [INFO] Running procedure cfg_text2git [INFO] == Command start (output follows) =====

UnsupportedOperation Traceback (most recent call last)

in ----> 1 datalad.api.create(path='mydataset', cfg_proc='text2git') /usr/local/lib/python3.6/dist-packages/datalad/interface/utils.py in eval_func(wrapped, instance, args, kwargs) 492 return results 493 lgr.log(2, "Returning return_func from eval_func for %s", wrapped_class) --> 494 return return_func(generator_func)(*args, **kwargs) 495 496 return eval_func(func) /usr/local/lib/python3.6/dist-packages/datalad/interface/utils.py in return_func(wrapped_, instance_, args_, kwargs_) 480 # unwind generator if there is one, this actually runs 481 # any processing --> 482 results = list(results) 483 # render summaries 484 if not result_xfm and result_renderer in ('tailored', 'default'): /usr/local/lib/python3.6/dist-packages/datalad/interface/utils.py in generator_func(*_args, **_kwargs) 411 result_log_level, 412 # let renderers get to see how a command was called --> 413 allkwargs): 414 for hook, spec in hooks.items(): 415 # run the hooks before we yield the result /usr/local/lib/python3.6/dist-packages/datalad/interface/utils.py in _process_results(results, cmd_class, on_failure, action_summary, incomplete_results, result_renderer, result_log_level, allkwargs) 550 render_n_repetitions = 10 if sys.stdout.isatty() else float("inf") 551 --> 552 for res in results: 553 if not res or 'action' not in res: 554 # XXX Yarik has to no clue on how to track the origin of the /usr/local/lib/python3.6/dist-packages/datalad/core/local/create.py in __call__(path, initopts, force, description, dataset, no_annex, annex, fake_dates, cfg_proc) 501 502 for cfg_proc_ in cfg_proc: --> 503 for r in tbds.run_procedure('cfg_' + cfg_proc_): 504 yield r 505 /usr/local/lib/python3.6/dist-packages/datalad/distribution/dataset.py in apply_func(wrapped, instance, args, kwargs) 501 elif i >= ds_index: 502 kwargs[orig_pos[i+1]] = args[i] --> 503 return f(**kwargs) 504 505 setattr(Dataset, name, apply_func(f)) /usr/local/lib/python3.6/dist-packages/datalad/interface/utils.py in eval_func(wrapped, instance, args, kwargs) 492 return results 493 lgr.log(2, "Returning return_func from eval_func for %s", wrapped_class) --> 494 return return_func(generator_func)(*args, **kwargs) 495 496 return eval_func(func) /usr/local/lib/python3.6/dist-packages/datalad/interface/utils.py in return_func(wrapped_, instance_, args_, kwargs_) 480 # unwind generator if there is one, this actually runs 481 # any processing --> 482 results = list(results) 483 # render summaries 484 if not result_xfm and result_renderer in ('tailored', 'default'): /usr/local/lib/python3.6/dist-packages/datalad/interface/utils.py in generator_func(*_args, **_kwargs) 411 result_log_level, 412 # let renderers get to see how a command was called --> 413 allkwargs): 414 for hook, spec in hooks.items(): 415 # run the hooks before we yield the result /usr/local/lib/python3.6/dist-packages/datalad/interface/utils.py in _process_results(results, cmd_class, on_failure, action_summary, incomplete_results, result_renderer, result_log_level, allkwargs) 550 render_n_repetitions = 10 if sys.stdout.isatty() else float("inf") 551 --> 552 for res in results: 553 if not res or 'action' not in res: 554 # XXX Yarik has to no clue on how to track the origin of the /usr/local/lib/python3.6/dist-packages/datalad/interface/run_procedure.py in __call__(spec, dataset, discover, help_proc) 450 # pass through here 451 on_failure='ignore', --> 452 return_type='generator' 453 ): 454 yield r /usr/local/lib/python3.6/dist-packages/datalad/interface/utils.py in generator_func(*_args, **_kwargs) 411 result_log_level, 412 # let renderers get to see how a command was called --> 413 allkwargs): 414 for hook, spec in hooks.items(): 415 # run the hooks before we yield the result /usr/local/lib/python3.6/dist-packages/datalad/interface/utils.py in _process_results(results, cmd_class, on_failure, action_summary, incomplete_results, result_renderer, result_log_level, allkwargs) 550 render_n_repetitions = 10 if sys.stdout.isatty() else float("inf") 551 --> 552 for res in results: 553 if not res or 'action' not in res: 554 # XXX Yarik has to no clue on how to track the origin of the /usr/local/lib/python3.6/dist-packages/datalad/core/local/run.py in __call__(cmd, dataset, inputs, outputs, expand, explicit, message, sidecar) 237 explicit=explicit, 238 message=message, --> 239 sidecar=sidecar): 240 yield r 241 /usr/local/lib/python3.6/dist-packages/datalad/core/local/run.py in run_command(cmd, dataset, inputs, outputs, expand, explicit, message, sidecar, extra_info, rerun_info, extra_inputs, rerun_outputs, inject, saver) 626 cmd_exitcode, exc = _execute_command( 627 cmd_expanded, pwd, --> 628 expected_exit=rerun_info.get("exit", 0) if rerun_info else None) 629 630 /usr/local/lib/python3.6/dist-packages/datalad/core/local/run.py in _execute_command(command, pwd, expected_exit) 468 log_stderr=False, 469 expect_stderr=True, --> 470 expect_fail=True, 471 # TODO stdin 472 ) /usr/local/lib/python3.6/dist-packages/datalad/cmd.py in run(self, cmd, log_stdout, log_stderr, log_online, expect_stderr, expect_fail, cwd, env, shell, stdin) 913 cwd=popen_cwd, 914 env=popen_env, --> 915 stdin=stdin) 916 917 except Exception as e: /usr/lib/python3.6/subprocess.py in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors) 685 (p2cread, p2cwrite, 686 c2pread, c2pwrite, --> 687 errread, errwrite) = self._get_handles(stdin, stdout, stderr) 688 689 # We wrap OS handles *before* launching the child, otherwise a /usr/lib/python3.6/subprocess.py in _get_handles(self, stdin, stdout, stderr) 1202 else: 1203 # Assuming file-like object -> 1204 c2pwrite = stdout.fileno() 1205 1206 if stderr is None: UnsupportedOperation: fileno #### Have you had any success using DataLad before? (to assess your expertise/prior luck. We would welcome your testimonial additions to https://github.com/datalad/datalad/wiki/Testimonials as well) Non python-api version works fine
bpoldrack commented 3 years ago

import datalad datalad.api.create(path='mydataset', cfg_proc='text2git')

Minor comment: Should likely be import datalad.api otherwise it should fail before execution.

At a first trial, this create call works for me in python 3.6, 3.7 and 3.8. So, it's not the python API per se leading to that issue. However, the error suggests that something goes wrong with the output capturing.

bpoldrack commented 3 years ago

@SteffenBrinckmann : Apart from the need to get to the core of the problem, just want to point out, that

import datalad.api
dataset = datalad.api.create('mydataset')
dataset.run_procedure('cfg_text2git')

Is supposed to do the same thing. Would be good to see, whether that works for you (and is a workaround in case it does).

SteffenBrinckmann commented 3 years ago

Sorry, it is an ipython/jupyter issue. If I run in native python it works fine.