mne-tools / mne-python

MNE: Magnetoencephalography (MEG) and Electroencephalography (EEG) in Python
https://mne.tools
BSD 3-Clause "New" or "Revised" License
2.72k stars 1.32k forks source link

Building docs on Windows #8027

Closed kalenkovich closed 4 years ago

kalenkovich commented 4 years ago

I have trouble building docs which I guess has to do with me being on Windows.

I've set up the environment according to the latest contributing guide:

pip/conda commands ```shell curl --remote-name https://raw.githubusercontent.com/mne-tools/mne-python/master/environment.yml conda env create --file environment.yml --name mnedev conda activate mnedev cd mne-python pip uninstall -y mne pip install -e . pip install -r requirements_doc.txt conda install -c conda-forge sphinx-autobuild doc8 ```

The html-dev-noplot target does give me the htmls that don't have the plots (as expected) but it breaks with an unifomative error after/during "embedding documentation hyperlinks for auto_tutorials..." step:

`make html-dev-noplot` output ```shell BUILD_DEV_HTML=1 sphinx-build -D plot_gallery=0 -b html -d _build/doctrees -nWT --keep-going . _build/html Running Sphinx v3.1.2 making output directory... done [autosummary] generating autosummary for: bibliography.rst, cited.rst, glossary.rst, index.rst, install\advanced.rst, install\contributing.rst, install\freesurfer.rst, install\index.rst, install\mne_c.rst, install\mne_python.rst, ..., overview\get_help.rst, overvie w\implementation.rst, overview\index.rst, overview\learn_python.rst, overview\matlab.rst, overview\migrating.rst, overview\roadmap.rst, python_reference.rst, references.rst, whats_new.rst [autosummary] generating autosummary for: D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.AcqParserFIF.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.Annotations.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo \doc\generated\mne.BaseEpochs.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.BiHemiLabel.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.Covariance.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\gene rated\mne.Dipole.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.DipoleFixed.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.Epochs.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.EpochsA rray.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.Evoked.rst, ..., D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.write_cov.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.write_events.rst , D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.write_evokeds.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.write_forward_solution.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.write_lab el.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.write_labels_to_annot.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.write_proj.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.write_s ource_spaces.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.write_surface.rst, D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\mne.write_trans.rst loading intersphinx inventory from https://docs.python.org/3/objects.inv... loading intersphinx inventory from https://numpy.org/devdocs/objects.inv... loading intersphinx inventory from https://scipy.github.io/devdocs/objects.inv... loading intersphinx inventory from https://matplotlib.org/objects.inv... loading intersphinx inventory from https://scikit-learn.org/stable/objects.inv... loading intersphinx inventory from https://numba.pydata.org/numba-doc/latest/objects.inv... loading intersphinx inventory from https://joblib.readthedocs.io/en/latest/objects.inv... loading intersphinx inventory from http://docs.enthought.com/mayavi/mayavi/objects.inv... loading intersphinx inventory from https://nipy.org/nibabel/objects.inv... loading intersphinx inventory from http://nilearn.github.io/objects.inv... loading intersphinx inventory from https://pysurfer.github.io/objects.inv... loading intersphinx inventory from https://pandas.pydata.org/pandas-docs/stable/objects.inv... loading intersphinx inventory from https://seaborn.pydata.org/objects.inv... loading intersphinx inventory from https://www.statsmodels.org/dev/objects.inv... loading intersphinx inventory from https://patsy.readthedocs.io/en/latest/objects.inv... loading intersphinx inventory from https://dipy.org/documentation/1.1.1./objects.inv/... loading intersphinx inventory from https://mne.tools/mne-realtime/objects.inv... loading intersphinx inventory from https://pierreablin.github.io/picard/objects.inv... generating gallery... WARNING: optipng binaries not found, PNG images and thumbnails will not be optimized Using pandoc version: None to convert rst text blocks to markdown for .ipynb files generating gallery for auto_examples\io... [100%] plot_read_proj.py trix.py generating gallery for auto_examples\simulation... [100%] plot_source_simulator.pyusing_subject_anatomy.py generating gallery for auto_examples\preprocessing... [100%] plot_xdawn_denoising.pyypy.py.py generating gallery for auto_examples\visualization... [100%] plot_xhemi.py ed.pytions.pypy generating gallery for auto_examples\time_frequency... [100%] plot_time_frequency_simulated.py_power.py generating gallery for auto_examples\stats... [100%] plot_sensor_regression.py st.py generating gallery for auto_examples\decoding... [100%] plot_receptive_field_mtrf.pyyatial_filter.py.py generating gallery for auto_examples\connectivity... [100%] plot_sensor_connectivity.py.pyivity.pyvolume.py generating gallery for auto_examples\forward... [100%] plot_source_space_morphing.pyource.py generating gallery for auto_examples\inverse... [100%] plot_vector_mne_solution.pyorm_inverse.pypy generating gallery for auto_examples\datasets... [100%] spm_faces_dataset.py generating gallery for auto_tutorials\intro... [100%] plot_50_configure_mne.py py generating gallery for auto_tutorials\io... [100%] plot_30_reading_fnirs_data.py generating gallery for auto_tutorials\raw... [100%] plot_40_visualize_raw.py generating gallery for auto_tutorials\preprocessing... [100%] plot_70_fnirs_processing.pyss.pypy generating gallery for auto_tutorials\epochs... [100%] plot_40_epochs_to_data_frame.py generating gallery for auto_tutorials\evoked... [100%] plot_whitened.py .py generating gallery for auto_tutorials\time-freq... [100%] plot_sensors_time_frequency.py generating gallery for auto_tutorials\source-modeling... [100%] plot_visualize_stc.py.pyate.pyion.py generating gallery for auto_tutorials\stats-sensor-space... [100%] plot_stats_spatio_temporal_cluster_sensors.pypy generating gallery for auto_tutorials\stats-source-space... [100%] plot_stats_cluster_time_frequency_repeated_measures_anova.pyy generating gallery for auto_tutorials\machine-learning... [100%] plot_sensors_decoding.py generating gallery for auto_tutorials\simulation... [100%] plot_point_spread.pytructures.py generating gallery for auto_tutorials\sample-datasets... [100%] plot_sleep.py BTi.pyom_elekta.py generating gallery for auto_tutorials\discussions... [100%] plot_background_statistics.py generating gallery for auto_tutorials\misc... [100%] plot_report.py e.py generating MNE command help ... [100%] mne_what.py checking for D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\references.bib in bibtex cache... not found parsing bibtex file D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\references.bib... parsed 159 entries building [mo]: targets for 0 po files that are out of date building [html]: targets for 28 source files that are out of date updating environment: [new config] 716 added, 0 changed, 0 removed checking for D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\references.bib in bibtex cache... not found_anovaa parsing bibtex file D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\references.bib... parsed 159 entries reading sources... [100%] whats_new tctionbution_repair D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\commands.rst:283: WARNING: undecodable source characters, replacing with "?": b' --rej-eeg=REJ_EEG EEG rejection parameter in >>>\xb5<<>>\xb5<<>>\xb5<<>>\xb5<<

And the "html_dev" target breaks almost immediately during "generating gallery...":

`make html_dev` output ```shell BUILD_DEV_HTML=1 sphinx-build -b html -d _build/doctrees -nWT --keep-going . _build/html Running Sphinx v3.1.2 loading pickled environment... done [autosummary] generating autosummary for: auto_examples\connectivity\plot_cwt_sensor_connectivity.rst, auto_examples\connectivity\plot_mixed_source_space_connectivity.rst, auto_examples\connectivity\plot_mne_inverse_coherence_epochs.rst, auto_examples\connectivity\ plot_mne_inverse_connectivity_spectrum.rst, auto_examples\connectivity\plot_mne_inverse_envelope_correlation.rst, auto_examples\connectivity\plot_mne_inverse_envelope_correlation_volume.rst, auto_examples\connectivity\plot_mne_inverse_label_connectivity.rst, auto_e xamples\connectivity\plot_mne_inverse_psi_visual.rst, auto_examples\connectivity\plot_sensor_connectivity.rst, auto_examples\datasets\plot_brainstorm_data.rst, ..., overview\get_help.rst, overview\implementation.rst, overview\index.rst, overview\learn_python.rst, o verview\matlab.rst, overview\migrating.rst, overview\roadmap.rst, python_reference.rst, references.rst, whats_new.rst generating gallery... Traceback (most recent call last): File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\psutil\_pswindows.py", line 679, in wrapper return fun(self, *args, **kwargs) File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\psutil\_pswindows.py", line 933, in create_time user, system, created = cext.proc_times(self.pid) ProcessLookupError: [Errno 3] No such process (originated from GetExitCodeProcess != STILL_ACTIVE) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\psutil\__init__.py", line 373, in _init self.create_time() File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\psutil\__init__.py", line 723, in create_time self._create_time = self._proc.create_time() File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\psutil\_pswindows.py", line 681, in wrapper raise convert_oserror(err, pid=self.pid, name=self._name) psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=18008) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx\events.py", line 110, in emit results.append(listener.handler(self.app, *args)) File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx_gallery\gen_gallery.py", line 381, in generate_gallery_rst gallery_conf = parse_config(app) File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx_gallery\gen_gallery.py", line 94, in parse_config gallery_conf = _complete_gallery_conf( File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx_gallery\gen_gallery.py", line 171, in _complete_gallery_conf gallery_conf['memory_base'] = _get_memory_base(gallery_conf) File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx_gallery\gen_rst.py", line 484, in _get_memory_base memories = memory_usage(proc, interval=1e-3, timeout=timeout) File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\memory_profiler.py", line 369, in memory_usage mem_usage = _get_memory( File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\memory_profiler.py", line 191, in _get_memory return tools[backend]() File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\memory_profiler.py", line 135, in ps_util_tool process = psutil.Process(pid) File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\psutil\__init__.py", line 346, in __init__ self._init(pid) File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\psutil\__init__.py", line 386, in _init raise NoSuchProcess(pid, None, msg) psutil.NoSuchProcess: psutil.NoSuchProcess no process found with pid 18008 The above exception was the direct cause of the following exception: Traceback (most recent call last): File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx\cmd\build.py", line 276, in build_main app = Sphinx(args.sourcedir, args.confdir, args.outputdir, File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx\application.py", line 277, in __init__ self._init_builder() File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx\application.py", line 333, in _init_builder self.events.emit('builder-inited') File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx\events.py", line 117, in emit raise ExtensionError(__("Handler %r for event %r threw an exception") % sphinx.errors.ExtensionError: Handler for event 'builder-inited' threw an exception Extension error: Handler for event 'builder-inited' threw an exception Exception ignored in: Traceback (most recent call last): File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\subprocess.py", line 942, in __del__ _warn("subprocess %s is still running" % self.pid, ResourceWarning: subprocess 18008 is still running make: *** [Makefile:44: html_dev] Error 2 ```

I don't understand how to debug this. Googling the errors didn't help.

Has anyone had similar issues with docs? Has anyone been able to build docs on Windows?

larsoner commented 4 years ago

There is no description here, but you should be able to build docs on windows in principle

kalenkovich commented 4 years ago

There is no description here...

I am sorry about this. Not sure what happened here: I still have the tab with the unsubmitted issue which I started filling yesterday. I will copy the description here now.

larsoner commented 4 years ago

Our doc build treats warnings as errors, so your errors are actually above:

D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\commands.rst:283: WARNING: undecodable source characters, replacing with "?": b'  --rej-eeg=REJ_EEG     EEG rejection parameter in >>>\xb5<<<V (peak to peak amplitude)\r'
D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\commands.rst:284: WARNING: undecodable source characters, replacing with "?": b'  --rej-eog=REJ_EOG     EOG rejection parameter in >>>\xb5<<<V (peak to peak amplitude)\r'
D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\commands.rst:357: WARNING: undecodable source characters, replacing with "?": b'  --rej-eeg=REJ_EEG     EEG rejection parameter in >>>\xb5<<<V (peak to peak amplitude)\r'
D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\generated\commands.rst:358: WARNING: undecodable source characters, replacing with "?": b'  --rej-eog=REJ_EOG     EOG rejection parameter in >>>\xb5<<<V (peak to peak amplitude)\r'

This suggests that it's a problem with encodings. Can you add this to doc/conf.py to see if it helps:

source_encoding = 'utf-8-sig'

It really shouldn't since that's the default. Maybe also a sphinx update might help?

kalenkovich commented 4 years ago

Thanks for taking a look, @larsoner!

I tried uncommenting # source_encoding = 'utf-8-sig' but it did not help, unfortunately.

Notepad++ says that "commands.rst" is actually ANSI-encoded. After checking several of the other files, it looks like all the other ones in "doc/generated/" are UTF-8 so changing source_encoding to ANSI would not work either. 😕

My sphinx (3.1.2) is up to date with the latest pypi version so no luck there either.

larsoner commented 4 years ago

Try changing this line:

https://github.com/mne-tools/mne-python/blob/master/doc/sphinxext/gen_commands.py#L69

to

    with open(out_fname, 'w', encoding='utf8') as f:

This should ensure that it's UTF8 encoded. It's possible there are other places that this needs to be done, too (probably in sphinx-gallery?). A workaround is to export PYTHONIOENCODING=utf8 before running, though we should probably fix any other occurrences, too.

kalenkovich commented 4 years ago

Setting the encoding for 'commands.rst' helped with the "undecodable source characters errors/warnings"!

html_dev-noplot still ends with "Errror 1" though.

Here is the new output ```bash BUILD_DEV_HTML=1 sphinx-build -D plot_gallery=0 -b html -d _build/doctrees -nWT --keep-going . _build/html Running Sphinx v3.1.2 loading pickled environment... done [autosummary] generating autosummary for: auto_examples\connectivity\plot_cwt_sensor_connectivity.rst, auto_examples\connectivity\plot_mixed_source_space_connectivity.rst, auto_examples\connectivity\plot_mne_inverse_coherence_epochs.rst, auto_examples\connectivity\ plot_mne_inverse_connectivity_spectrum.rst, auto_examples\connectivity\plot_mne_inverse_envelope_correlation.rst, auto_examples\connectivity\plot_mne_inverse_envelope_correlation_volume.rst, auto_examples\connectivity\plot_mne_inverse_label_connectivity.rst, auto_e xamples\connectivity\plot_mne_inverse_psi_visual.rst, auto_examples\connectivity\plot_sensor_connectivity.rst, auto_examples\datasets\plot_brainstorm_data.rst, ..., overview\get_help.rst, overview\implementation.rst, overview\index.rst, overview\learn_python.rst, o verview\matlab.rst, overview\migrating.rst, overview\roadmap.rst, python_reference.rst, references.rst, whats_new.rst generating gallery... WARNING: optipng binaries not found, PNG images and thumbnails will not be optimized Using pandoc version: None to convert rst text blocks to markdown for .ipynb files generating gallery for auto_examples\io... [100%] plot_read_proj.py trix.py generating gallery for auto_examples\simulation... [100%] plot_source_simulator.pyusing_subject_anatomy.py generating gallery for auto_examples\preprocessing... [100%] plot_xdawn_denoising.pyypy.py.py generating gallery for auto_examples\visualization... [100%] plot_xhemi.py ed.pytions.pypy generating gallery for auto_examples\time_frequency... [100%] plot_time_frequency_simulated.py_power.py generating gallery for auto_examples\stats... [100%] plot_sensor_regression.py st.py generating gallery for auto_examples\decoding... [100%] plot_receptive_field_mtrf.pyyatial_filter.py.py generating gallery for auto_examples\connectivity... [100%] plot_sensor_connectivity.py.pyivity.pyvolume.py generating gallery for auto_examples\forward... [100%] plot_source_space_morphing.pyource.py generating gallery for auto_examples\inverse... [100%] plot_vector_mne_solution.pyorm_inverse.pypy generating gallery for auto_examples\datasets... [100%] spm_faces_dataset.py generating gallery for auto_tutorials\intro... [100%] plot_50_configure_mne.py py generating gallery for auto_tutorials\io... [100%] plot_30_reading_fnirs_data.py generating gallery for auto_tutorials\raw... [100%] plot_40_visualize_raw.py generating gallery for auto_tutorials\preprocessing... [100%] plot_70_fnirs_processing.pyss.pypy generating gallery for auto_tutorials\epochs... [100%] plot_40_epochs_to_data_frame.py generating gallery for auto_tutorials\evoked... [100%] plot_whitened.py .py generating gallery for auto_tutorials\time-freq... [100%] plot_sensors_time_frequency.py generating gallery for auto_tutorials\source-modeling... [100%] plot_visualize_stc.py.pyate.pyion.py generating gallery for auto_tutorials\stats-sensor-space... [100%] plot_stats_spatio_temporal_cluster_sensors.pypy generating gallery for auto_tutorials\stats-source-space... [100%] plot_stats_cluster_time_frequency_repeated_measures_anova.pyy generating gallery for auto_tutorials\machine-learning... [100%] plot_sensors_decoding.py generating gallery for auto_tutorials\simulation... [100%] plot_point_spread.pytructures.py generating gallery for auto_tutorials\sample-datasets... [100%] plot_sleep.py BTi.pyom_elekta.py generating gallery for auto_tutorials\discussions... [100%] plot_background_statistics.py generating gallery for auto_tutorials\misc... [100%] plot_report.py e.py generating MNE command help ... [100%] mne_what.py checking for D:\onedrives\OneDrive\projects\ssvep_tutorial\repo\doc\references.bib in bibtex cache... up to date building [mo]: targets for 0 po files that are out of date building [html]: targets for 1 source files that are out of date updating environment: 0 added, 6 changed, 0 removed reading sources... [100%] generated/mne.write_surface looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] python_reference generating indices... genindex py-modindexdone writing additional pages... searchdone copying images... [100%] auto_examples/inverse/images/thumb/sphx_glr_plot_vector_mne_solution_thumb.pngerse_thumb.pngngpngngpngsures_anova_thumb.pngg copying downloadable files... [100%] auto_tutorials/auto_tutorials_jupyter.zip _frequency.ipynbfrequency_repeated_measures_anova.ipynbb copying static files... ... done copying extra files... done dumping search index in English (code: en)... done dumping object inventory... done build finished with problems, 1 warning. Sphinx-gallery gallery_conf["plot_gallery"] was False, so no examples were executed. embedding documentation hyperlinks... embedding documentation hyperlinks for auto_examples... [100%] plot_xhemi.html ized.htmlons.htmlmlwer.htmlmy.html embedding documentation hyperlinks for auto_tutorials... [100%] plot_sensors_time_frequency.htmly_repeated_measures_anova.htmll make: *** [Makefile:54: html_dev-noplot] Error 1 ```

Near the end, it says "build finished with problems, 1 warning.". The only warning I could find is this:

WARNING: optipng binaries not found, PNG images and thumbnails will not be optimized

Could it be that everything is actually building correctly and the error is the consequence of this warning?

larsoner commented 4 years ago

Yes, warnings are treated as errors so this is indeed the cause of the failure. Can you see how hard it is to install optipng? If it's difficult on windows, we can make that fix build conf entry depend on the OS.

kalenkovich commented 4 years ago

I download "optipng.exe" here and saved it in the "doc" folder. This got rid of the optipng warning. The new one is

WARNING: dot command 'dot' cannot be run (needed for graphviz output), check the graphviz_dot setting

Googling this one now 😃

larsoner commented 4 years ago

You need to have the graphviz command-line programs on your machine.

BTW once you figure this out, it would be nice to add this info to the dev docs for the next person. The requirements are valid on all systems -- on macOS you probably use brew, Linux your package manager, and Windows maybe we should give people links to download pages.

kalenkovich commented 4 years ago

I installed graphviz using conda. The html_dev-noplot finished without errors!

The html_dev still breaks though:

make html_dev ``` BUILD_DEV_HTML=1 sphinx-build -b html -d _build/doctrees -nWT --keep-going . _build/html Running Sphinx v3.1.2 loading pickled environment... done [autosummary] generating autosummary for: auto_examples\connectivity\plot_cwt_sensor_connectivity.rst, auto_examples\connectivity\plot_mixed_source_space_connectivity.rst, auto_examples\connectivity\plot_mne_inverse_coherence_epochs.rst, auto_examples\connectivity\ plot_mne_inverse_connectivity_spectrum.rst, auto_examples\connectivity\plot_mne_inverse_envelope_correlation.rst, auto_examples\connectivity\plot_mne_inverse_envelope_correlation_volume.rst, auto_examples\connectivity\plot_mne_inverse_label_connectivity.rst, auto_e xamples\connectivity\plot_mne_inverse_psi_visual.rst, auto_examples\connectivity\plot_sensor_connectivity.rst, auto_examples\datasets\plot_brainstorm_data.rst, ..., overview\get_help.rst, overview\implementation.rst, overview\index.rst, overview\learn_python.rst, o verview\matlab.rst, overview\migrating.rst, overview\roadmap.rst, python_reference.rst, references.rst, whats_new.rst generating gallery... Traceback (most recent call last): File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\psutil\_pswindows.py", line 679, in wrapper return fun(self, *args, **kwargs) File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\psutil\_pswindows.py", line 933, in create_time user, system, created = cext.proc_times(self.pid) ProcessLookupError: [Errno 3] No such process (originated from GetExitCodeProcess != STILL_ACTIVE) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\psutil\__init__.py", line 373, in _init self.create_time() File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\psutil\__init__.py", line 723, in create_time self._create_time = self._proc.create_time() File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\psutil\_pswindows.py", line 681, in wrapper raise convert_oserror(err, pid=self.pid, name=self._name) psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=14508) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx\events.py", line 110, in emit results.append(listener.handler(self.app, *args)) File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx_gallery\gen_gallery.py", line 381, in generate_gallery_rst gallery_conf = parse_config(app) File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx_gallery\gen_gallery.py", line 94, in parse_config gallery_conf = _complete_gallery_conf( File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx_gallery\gen_gallery.py", line 171, in _complete_gallery_conf gallery_conf['memory_base'] = _get_memory_base(gallery_conf) File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx_gallery\gen_rst.py", line 484, in _get_memory_base memories = memory_usage(proc, interval=1e-3, timeout=timeout) File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\memory_profiler.py", line 369, in memory_usage mem_usage = _get_memory( File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\memory_profiler.py", line 191, in _get_memory return tools[backend]() File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\memory_profiler.py", line 135, in ps_util_tool process = psutil.Process(pid) File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\psutil\__init__.py", line 346, in __init__ self._init(pid) File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\psutil\__init__.py", line 386, in _init raise NoSuchProcess(pid, None, msg) psutil.NoSuchProcess: psutil.NoSuchProcess no process found with pid 14508 The above exception was the direct cause of the following exception: Traceback (most recent call last): File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx\cmd\build.py", line 276, in build_main app = Sphinx(args.sourcedir, args.confdir, args.outputdir, File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx\application.py", line 277, in __init__ self._init_builder() File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx\application.py", line 333, in _init_builder self.events.emit('builder-inited') File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\site-packages\sphinx\events.py", line 117, in emit raise ExtensionError(__("Handler %r for event %r threw an exception") % sphinx.errors.ExtensionError: Handler for event 'builder-inited' threw an exception Extension error: Handler for event 'builder-inited' threw an exception Exception ignored in: Traceback (most recent call last): File "d:\software_windows\miniconda3\envs\mne_ssvep_tutorial\lib\subprocess.py", line 942, in __del__ _warn("subprocess %s is still running" % self.pid, ResourceWarning: subprocess 14508 is still running make: *** [Makefile:44: html_dev] Error 2 ```

it would be nice to add this info to the dev docs for the next person

Sure thing! Just to recap what has helped with the html_dev-noplot target:

drammock commented 4 years ago

Specifying encoding='utf-8' is something we could change for all users, rather than asking windows users to change it locally. Istalling graphviz from conda is also something we could do for all platforms. Not sure about optipng, providing a download link maybe be the best option there.

larsoner commented 4 years ago

@kalenkovich can you open a PR to update the gen_commands.py file with the encoding line?

For the graphviz and optipng it would be good to update the dev docs with platform-agnostic (graphviz) and platform-specific (optipng) instructions. Up for changing that in the same PR?

kalenkovich commented 4 years ago

Up for changing that in the same PR?

Sure. But maybe we should figure out why the docs are not building first? Or, the partial solution is ok?

larsoner commented 4 years ago

If you installed memory_profiler recently and it's giving these problems then it's probably just unstable on Windows. We should change this:

    'show_memory': True,

to

    'show_memory': not sys.platform.startswith('win'),
kalenkovich commented 4 years ago

@larsoner, using 'show_memory': not sys.platform.startswith('win') helped with breaking on the "generating gallery" step! How did you figure this one out? 😲

The changes to docs, conf.py, gen_commands.py, etc. that we discussed above - do they belong to "Changelog", "Bug", or "API"?

I also have a new problem (I am good at getting those) make html_dev gets stuck on some files (plot_movement_detection.py, plot_muscle_detection.py). I've been deleting these and rerunning make html_dev to figure out the full list of problematic files. I'll report back here once I get to the end of html_dev or to a different kind of problem.

larsoner commented 4 years ago

@larsoner, using 'show_memory': not sys.platform.startswith('win') helped with breaking on the "generating gallery" step! How did you figure this one out? 

The errors you were getting had to do with psutil trying to measure memory. It has been unreliable on macOS before, so it makes sense it would be even less reliable on Windows. So it makes sense to disable it -- it's helpful but not (usually) critical information.

The changes to docs, conf.py, gen_commands.py, etc. that we discussed above - do they belong to "Changelog", "Bug", or "API"?

Changelog, something like "- Improved documentation building instructions and execution on Windows by ..." would be good

I also have a new problem (I am good at getting those) make html_dev gets stuck on some files (plot_movement_detection.py, plot_muscle_detection.py)

Looking briefly at plot_muscle_detection.py I don't see why this would be problematic. Even looking at the semi-private sg_execution_times for when CircleCI built the docs, it only took 400 MB and 6 sec. I'll boot into my Windows machine to take a look.

Does it fail if you do just PATTERN=muscle_detection make html_dev-pattern?

larsoner commented 4 years ago

The changes to docs, conf.py, gen_commands.py, etc. that we discussed above - do they belong to "Changelog", "Bug", or "API"?

... before I boot into my Windows VM, can you open a WIP PR with these changes already in it so I can just checkout your branch?

kalenkovich commented 4 years ago

Sure thing! Give me a couple of minutes.

kalenkovich commented 4 years ago

Here it is: https://github.com/mne-tools/mne-python/pull/8057

marsipu commented 4 years ago

Thanks to the updated contribution-guide, I could finally build the docs on Windows too. But I had to overcome some additional problems which were not described:

Should we add them too? (@kalenkovich how did you solve them?)

kalenkovich commented 4 years ago

Hi, I’m on vacation for the next two weeks so I won’t be able to look into this in detail now. What I can tell you now is that I had installed make in the base conda environment some time ago - I think this option should be added to the guide, probably installing into the mne-dev environment instead of the base one. The thing is, I didn’t need to change anything else so I must have installed something else at one time or another but I have no idea what it was. It might have been a conda-less install of Cygwin or something like that - unfortunately, I can’t tell what it was right now. It would be great to try to build the docs on a fresh install of Windows 10 to figure it out. Hope this helps.

On Fri, 31 Jul 2020 at 21:33, Martin Schulz notifications@github.com wrote:

Thanks to the updated contribution-guide, I could finally build the docs on Windows too. But I had to overcome some additional problems which were not described:

  • Install GNU-Make (I did it via chocolatey)
  • Replace linux-related lines in Makefile (e.g. BUILD_DEV_HTML=1 for docs and rm -f .coverage for tests, ...)

Should we add them too? (@kalenkovich https://github.com/kalenkovich how did you solve them?)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mne-tools/mne-python/issues/8027#issuecomment-667290685, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACGUAWSSGLB5NSZECHHR3NTR6MFATANCNFSM4PBJ25LQ .

marsipu commented 4 years ago

Thank you, I will try it again with the conda version, maybe that is also a solution for the MakeFiles. Otherwise, we maybe need Cygwin as a requirement or MakeFiles adapted to Windows.

I tried the WSL2 too, but I ran into some OpenGL-issues with the XMing-Server when I tried to run the documentation

marsipu commented 4 years ago

Installing make with conda-forge worked but didn't fix the problem with the unix-commands in the Makefile. If I replace for example BUILD_DEV_HTML=1 $(SPHINXBUILD) [...] with set BUILD_DEV_HTML=1 $(SPHINXBUILD) [...] in the Doc-Makefile, it works.

I installed Cygwin too and tried to run a Makefile with it, but I still have problems with activating my conda environment, which I couldn't resolve yet (conda not initialized).

What do you think would be the best option to recommend users for Windows&Make?

  1. Installing make via conda-forge and creating a seperate Makefile for Windows
  2. As 1, but with one Makefile recognizing the OS like in this example
  3. Installing Cygwin and running make from the Cygwin-Terminal
larsoner commented 4 years ago

IIRC I just use msysgit that comes with git for windows and add make. Solves two problems: getting git and being able to use BASH / standard Makefile. Have you tried this?

marsipu commented 4 years ago

@larsoner Thank you, I tried it and put the make.exe from stackoverflow into Git/cmd/. Unfortunately, for me it says "msys-1.0.dll could not be found" and doesn't work. I also tried this with the same result

agramfort commented 4 years ago

Sphinx does not require make on windows. We should see how our makefile hacks can be used without make

larsoner commented 4 years ago

Unfortunately, for me it says "msys-1.0.dll could not be found" and doesn't work. I also tried this with the same result

This is while in the git bash terminal? If so I'll try uninstalling and reinstalling with step-by-step instructions

Sphinx does not require make on windows. We should see how our makefile hacks can be used without make

No idea if there is some tool to convert Makefile style commands to something else. Somehow I doubt it because this would just replicate make itself...

Also there's value in getting people a usable bash shell on Windows, so I'd rather go that route. It would allow us to assume all devs have access to some basic bash utilities, which is nice. Maybe Windows Subsystem for Linux would work here.

marsipu commented 4 years ago

This is while in the git bash terminal?

Oh no, I only tried it with the Anaconda-Terminal. In the Git Bash-Terminal for Windows it works fine (after once conda init bash and activating my dev-environment, which didn't work in Cygwin for me). This works without modifying the Makefile, so the only thing to do for new contributors on Windows is to download make (maybe this one got the more reliable download-source and instructions?)

Should I add something for this into the documentation?

Maybe Windows Subsystem for Linux would work here

I tried it with WSL2 and Xming-Server, but got a OpenGl-Error when I tried building the docs.

qt.qpa.xcb: X server does not support XInput 2
failed to get the current screen resources
2020-08-24 17:57:41.050 (   1.447s) [         9DC0740]vtkXOpenGLRenderWindow.:251    ERR| vtkXOpenGLRenderWindow (0x7fffbccbdf40): Could not find a decent config

2020-08-24 17:57:41.051 (   1.448s) [         9DC0740]vtkXOpenGLRenderWindow.:468    ERR| vtkXOpenGLRenderWindow (0x7fffbccbdf40): Could not find a decent visual

For other things with no graphical output involved, like mne make_watershed or recon-all, WSL2 works fine for me.

larsoner commented 4 years ago

If WSL doesn't work for graphics then let's not recommend it. If msysgit + make download/install works let's just give instructions for that. I've used it for years without problems, and it does take care of git + make handling in one solution, which is nice.

@marsipu want to take a stab at updating the docs as was done in #8057 ? It would be great to get these docs updated in the next couple of weeks so that it can be part of the upcoming 0.21 release

marsipu commented 4 years ago

want to take a stab at updating the docs as was done in #8057 ?

Yes sure, I'll do it in the next days

marsipu commented 4 years ago

Short question for this passage:

  • On Windows, we recommend git Bash rather than the official Windows version of git, because git Bash provides its own shell that includes many Linux-equivalent command line programs that are useful for development. Windows 10 also offers the Windows subsystem for Linux_ that offers similar functionality to git Bash, but has not been widely tested by MNE-Python developers yet.

The downloads from official git and git for Windows for Windows are the same. If there is no other major difference, could I shorten it to avoid confusion?

larsoner commented 4 years ago

Yeah feel free to update or shorten

marsipu commented 4 years ago

May I add myself to names.inc?

agramfort commented 4 years ago

sure !

larsoner commented 4 years ago

Yes typically we update latest.inc to mention the improvement and the name at the end needs a corresponding entry in names.inc

marsipu commented 4 years ago

Cool, thank you. Very exciting:)

larsoner commented 4 years ago

Fixed by #8164