What is the future of Rodeo?

[ghost]

This is an issue currently facing by many users of Rodeo, so please dont close this

I have been a user of rodeo for an year or more. Eventhough rodeo has many bugs and problems I stayed with it because of my love for RStudio and belived Rodeo can become the "RStudio" of python. The biggest problem for me in rodeo is its memory and over heating problems and the culprit I think is electron.

For a year there seems to be no update or development happening in this project. Everything is breaking in rodeo ide, forum had been removed and there is problems with the rodeo website itself. As an user of this software, I want to know whether this project is abandoned. Can some one from yhat give some details about rodeo's developmen? What is happening there?

[abalter]

I second the question.

[m-macaskill]

The Yhat founders moved on after an acqusition by Alteryx in June 2017: http://blog.yhat.com/posts/alteryx-acquires-yhat.html

and their other technology is apparently being readily adopted there: https://techcrunch.com/2017/09/12/alteryx-promote-puts-data-science-to-work-across-the-company/

As no pull request to this project has been merged since Jan 2017, I think it is reasonably clear that it has been abandoned in this form at least. It is open-source, of course, so development could continue from anyone else who wishes to contribute. But a strong community of contributors other than the founders has not developed: https://github.com/yhat/rodeo/graphs/contributors

so that doesn't give me much confidence that the project will continue, barring the entry of a new enthusiastic and committed developer(s). Such people wouldn't necessarily have administrative rights to this particular repository however, so it would probably need to be forked.

This is a shame, since Rodeo was a very nice implementation of a scientific Python IDE. But as it definitely requires further, and ongoing, development, I think it would be unwise to invest time and energy in trying to keep running it as an end-user. Existing performance and bug issues will only increase over time, and new or improved feature developments are not likely.

[TakenPilot]

☝️ This is true. And since I worked for less than a year at Yhat, I moved on to another company when that was happening.

If someone does pick up this project, that would be amazing and I would gladly walk them through the code or help them out.

[m-macaskill]

That's a great offer, Dane, I hope someone will take it up. So much was achieved in this very nice project and it would be an amazing asset to the Python community if it's development could be continued, in the same way that RStudio has gone such a long way to making R accessible to a broad audience, but with an even more clean look and feel.

[emprovements]

I am total noob with all JavaScript related stuff but it is very sad that project like this, which I think is the most cleanest and nicest looking IDE for python data science I have seen is slowly dieing.

[TakenPilot]

I could pick it up again if I get permission from @glamp, but otherwise it would feel weird. 🙂

[emprovements]

I will try to contribute as well when I gain some JS knowledge.

[abalter]

I have pretty good JavaScript skills, and I'm a solid programmer. But I have no experience with large software projects. What would be the next task? I'd be happy to take a stab at it.

[abalter]

Honestly, I wonder if the best thing would be to start from scratch. JupyterLab has some of the functionality already. It is built on a JS library called Phosphor that is designed to create panels and menus etc. Jupyter is already developing a variable explorer.

Also, the Eclipse project is abandoning Che and Orion in favor of developing an IDE meant to work on both cloud and desktop called Theia. They are using Phosphor for this project. Between that and Jupyter, this suggests Phosphor will have a long well-supported life.

However, Phosphor has very little recent activity. Also, unfortunately, there is almost no documentation for how to get started using Phosphor :(

On the other hand, there is also a panel layout library called Golden-Layout that is well documented and looks super sharp and useful. It is very active with 45 contributors and lots of recent commits. But no guarantees on its future.

[asampat3090]

Is there another similar project that is being developed? Basically it seems like all of the users of Rodeo are primarily looking for an "RStudio" for Python. Seems like RStudio is currently natively built for each platform but not sure where it started. I wonder if it would be good to start with one OS and go from there. Thoughts? @abalter @potholiday

If it makes most sense to continue with the development here, I would be glad to learn from you @TakenPilot, this project is pretty amazing and I've been a big fan for a while.

[ghost]

I also agree with you on starting with one OS. I love RStuido because its really fast and doesn't clog memory when working with big data. That's probably because of it being natively built for each platforms. Whereas most python IDE's which I have used including Rodeo clogs memory and slows the computer down. I would prefer Rodeo rebuilt using c or c++ but it would require a lot of time and hard work. In my knowledge there is no good alternative for RStudio in python and so I have moved to lower level IDE's such as vim. Its hard to work with big data in it but it doesn't clog my memory.

[m-macaskill]

RStudio development occurs cross-platform, as described here:

https://github.com/rstudio/rstudio/wiki/RStudio-Development

[ghost]

I could pick it up again if I get permission from @glamp, but otherwise it would feel weird. 🙂

If that's so, please let me know. I'm in for a rebrand also.

[abalter]

@potholiday -- how do you feel about Atom and VS Code which are built on electron?

[xguse]

@asampat3090 you might take a look at Spyder. It ships with anaconda and also aims to provide a similar environment to rstudio. https://www.spyder-ide.org/

[boralprophecy]

In for development. This is a really good project and it will help millions of people if done.

[CAM-Gerlach]

As an obligatory plug, Spyder seems to be a clear upgrade path for Rodeo users, IMO. As a huge R/Rstudio fan who moved to Python around a year and a half ago (right after Rodeo had reached the peak of its development), I actually waffled between choosing it and Rodeo due to the former's similarity to my favorite IDE vs. the latter's greater maturity and features, I don't regret my final decision. The fantastic thing about Spyder is its written in pure Python—the entire IDE is coded in the very same language its primarily used for, and in fact Spyder itself is primarily developed in Spyder—and it uses the Qt Framework as its GUI (C++, though its Python bindings), so it looks like any other native application. Of course, I'm a little biased since i liked it so much, I became one of the developers myself recently, since its a 100% community developed project. The pace of development has continued to increase over the years since its first release in 2009, and our community of hundreds of thousands of users on every continent continues to grow along with our number of contributors.

Its got a essentially a strict superset of the features and functionality in Rodeo and is at parity or even most sophisticated than Rstudio in most respects, with an advanced editor with all the core features commonly found in other mainstream IDEs and editors (we're currently integrating the same completion/introspection/analysis architecture as Atom and VSCode for Spyder 4, to replace our in-house one); a Console that supports an unlimited number of tabs/sessions and can launch and connect to kernels in any local or remote Python installation and that has IPython, Matplotlib, Cython, Sympy and Pylab support built right in, a Help pane that interactively renders rich documentation for both your own and external objects on demand, a file explorer, project system, outline viewer, interactive debugger, history log, static code analyzer, online help browser, GUI data import tool that supports reading a variety of filetypes and can convert to lists, numpy arrays or pandas dataframes, session save and restore, a new plot module for Spyder 4 inspired by Rodeo's own UI, and of course our marquee feature, the Variable Explorer, with full support for interactive viewing, editing, manipulation and visualization of not only scalar types, lists/dicts/tuples/sets, 1/2/3D numpy arrays, Pandas dataframes, Pillow images and more, but also functions, modules, classes and arbitrary Python objects.

We also have a growing ecosystem of first and third-party plugins which should grow and get much easier to make your own (all in Python, of course) with the major refactoring and public API coming in Spyder 4; officially supported examples include full Jupyter notebook integration inside Spyder, an R Markdown/knitr/sweave equivalent, a cross-platform system shell plugin for the console, a Vim plugin that emulates its editing functionality and shortcuts, line and memory profilers, a unit tests plugin supporting all the popular frameworks (Pytest, Unittest and nose), a GUI package and environment manager (the basis for Anaconda Navigator, in fact), and an plugin to automatically PEP8-ify your code. Active development on most of them is temporarily paused to focus on Spyder 4, which itself has a large number of new features and improvements as well as a major rework to the plugin system and API, but should resume when that goes final in the next couple months. We're active on Gitter, Stack Exchange, Github and our Google Groups mailing list as well as social media, so feel free to ask if you have questions.

[OldGuyInTheClub]

Does it have dockable plots and/or tabs for figures? The last time I tried Spyder I couldn't get past plot windows disappearing when using the Spyder GUI and the Spyder GUI did not work well when tiled with plot windows.

[CAM-Gerlach]

Does it have dockable plots and/or tabs for figures?

It currently has the option for displaying plots inline in the Console as well as standard plot windows, and a full Plot pane built into the UI (that can be moved around, resized and docked at will, like any other pane) is already implemented in Spyder 4 and available in the released Beta (Spyder 4 goes final in a couple months). The latter is actually directly inspired by Rodeo's Plot viewer, with some added features, customization options and UI functionality. Here's a quick preview:

[OldGuyInTheClub]

Thanks. Is any interactivity possible either with inline plots or this pane? One of the many things I like about Matlab's and RStudio's UIs is the ability to keep code, variables, plots, debugger, etc. in view. Rodeo was close but didn't have the debugger.

[CAM-Gerlach]

Is any interactivity possible either with inline plots or this pane?

Not with inline plots (by design), as they are intended as immutable output. Its rather limited with the pane at the moment, just zooming, panning, and showing/hiding an outline/border. However, more can likely be added in the future; given that Spyder is written in pure Python, the same language you use it for, and its open-source and developed entirely by and for the scientific Python community. You can also easily swap the backend to "Automatic" in preferences to open the plot in the dedicated viewer with expanded functionality (e.g. interacting with 3D plots); we might be able to add an option to do that directly from the plot pane for a given plot.

One of the many things I like about Matlab's and RStudio's UIs is the ability to keep code, variables, plots, debugger, etc. in view.

Yeah, I really like that. I dunno about MATLAB but Spyder's UI is much more customizable than Rstudio's limited one; you can open multiple vertical and horizontally split editor panes, can move around any pane to any area of the screen, and customize the layout, size and position nearly infinitely to show as much or as little as you want. For example, there's a number of built-in window layout presets to match Rstudio or MATLAB:

and you can save, customize and reset your own:

(These are all older versions of Spyder 3)

Rodeo was close but didn't have the debugger.

At least with regard to viewing the current state, Spyder's Variable Explorer is a lot like an always-on debugger on steriods; it when I first tried it, it really blew Rstudio's viewer out of the water. Not only can you recursively inspect collections, functions, modules, and arbitrary objects, but you can actually edit most of them in memory, including specialized viewers to interact with arrays, dataframes, lists/dicts/sets/tuples, images and more.

Currently, the GUI and Console is integrated with ipdb for the debugger, but we're currently finishing the implementation of own powerful, full-featured debugging kernel similar to that found in heavy-duty IDEs to make it vastly more capable, along with UI/UX improvements.

[OldGuyInTheClub]

Thank you. The last time I tried it was a couple of years ago (Rev. 2?) and had problems with the debugger crashing along with the difficulty in swapping back and forth between the GUI and plot windows. I will give Spyder 3 a try and check out the new features. The Rev. 4 debugger sounds very interesting. Out of curiosity, what happens in the console with, say, Bokeh plots? Would they retain their interactivity?

[CAM-Gerlach]

Out of curiosity, what happens in the console with, say, Bokeh plots? Would they retain their interactivity?

My guess is maybe not, if they are HTML/CSS/JS based, but I really don't know for sure—you might be able to set the backend properly so they work.

[OldGuyInTheClub]

Spyder 3 feels better than the prior version I tried. The upcoming changes are intriguing and I look forward to them. I am sorry to hear about the situation with Anaconda. I made a small donation to the cause. I'll start tracking the Spyder github page(s) for further support.

[versipellis]

It's also worth throwing it out there, Pycharms is worth it and more featureful if you don't mind forking out some money.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/yhat/rodeo/issues/655#issuecomment-443394530, or mute the thread https://github.com/notifications/unsubscribe-auth/AGRjWnscS4uuDn0sek-qQ8HFtR7ZZpOqks5u0fKagaJpZM4Saxvo .

[OldGuyInTheClub]

I think there's a difference between tools for software development and tools for technical computing/algorithm exploration/data analysis. I can see PyCharm's power for the former but my needs are in the latter camp.

[OldGuyInTheClub]

Out of curiosity, what happens in the console with, say, Bokeh plots? Would they retain their interactivity?

My guess is maybe not, if they are HTML/CSS/JS based, but I really don't know for sure—you might be able to set the backend properly so they work.

I installed the Jupyter Notebook plugin and tried a simple x-y demo Bokeh plot. It rendered with its interactive tools. Panning, zooming, and reset all worked.

[versipellis]

I'm a data scientist by trade- my work falls firmly in the latter camp. PyCharm has Jupyter from its interface, too. I'm on mobile, so I'm not going to be able to wax poetic, but take a look at it's featureset again. I moved on to it from Rodeo, and I find it even more powerful than RStudio.

On Nov 30, 2018 22:41, "Old Guy in the Club" notifications@github.com wrote:

I think there's a difference between tools for software development and tools for technical computing/algorithm exploration/data analysis. I can see PyCharm's power for the former but my needs are in the latter camp.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/yhat/rodeo/issues/655#issuecomment-443396331, or mute the thread https://github.com/notifications/unsubscribe-auth/AGRjWkiyfjcHYXeW-hvRL2qzR3C4QQPJks5u0fqEgaJpZM4Saxvo .

[OldGuyInTheClub]

Ok, I can check it out again.

[CAM-Gerlach]

I am sorry to hear about the situation with Anaconda.

Thanks so much for your support! Actually, Anaconda's funding ended over a year ago, and Spyder's users picked up much of the slack in donations and we've also gotten grants and other funding from Python community donations to help accelerate Spyder 4 development, and in fact we've kept up a pretty consistent level of code commits and substantially increasing number of new contributors. Spyder's been a community project long before Anaconda even existed, developed primarily or entirely by volunteers, but your support helps specifically fund students and junior developers to work on specific Spyder features most requested by the user community. That's what really makes Spyder special—unlike other IDEs created by big companies primarily for paying customers, Spyder is developed and used by and for its users all around the world.

I think there's a difference between tools for software development and tools for technical computing/algorithm exploration/data analysis.

Very true. Spyder's a big step over Rodeo in the former regard, and Spyder 4 will bring a number of major improvements to help close the gap with those tools in many areas, but if I was doing hardcore software development I'd likely still use PyCharm or PyDev, because they are specifically built for that usecase. However, conversely, while they're certainly usable to develop scientific packages, they simply aren't designed for interactive data exploration, analysis, visualization, and incremental execution like Spyder is, nor nearly as integrated with all the core packages and infrastructure of the PyData ecosystem. Pycharm is a great tool, but for the great majority of users moving from Rodeo to Pycharm would be like a repair technician switching from a pickup to a semi truck, as opposed to a specialized work van—the former more powerful overall, but the latter more suited for the task at hand.

@versipellis If you don't mind sharing, which particular areas in your data science work do you find Spyder lacking in relative to PyCharm (and similar "big-iron" software development ? The two biggest areas of improvement we've heard from our many users so far is its debugger and its code completion (which its why those are two of the biggest features prioritized for Spyder 4), but we always really appreciate the feedback of an experienced data science professional like yourself and as a community-led project, we actually really do take it into account to prioritize our limited resources to where our users need them most. Thanks!

[OldGuyInTheClub]

Good perspectives. Where's the best place for the new/occasional Spyder user to get questions answered? I am coming up with a few on general use and also the relationship between Spyder and other projects like Jupyter. I don't think they're right for the Spyder Issues pages. I normally do thorough searches before posting to other sites like the Stack* hierarchy but have had bad luck with them.

[CAM-Gerlach]

Where's the best place for the new/occasional Spyder user to get questions answered?

As you seem to have already found out on your own, our Gitter is the prefect place for those sorts of questions.

[OldGuyInTheClub]

Yep, lucky to have stumbled across it.

[medmatix]

There is a trend I see fighting non-corporate models of organization which I wish to share. It is something that leaves me frustrated but more so in the current 'anti-social' US political climate.

Mutual Companies and opensource based organizations are being converted or acquired in such a manner as to be disappearing from the organizational spectrum.

Witness Microsoft's acquisition of Github, Proprietary oriented Oracle's acquisition of opensource friendly SUN Microsystems, Alteryx's acquisition of yhat and somewhat more frightening (to me) DARPA's removal of further open develoment of the Epiphany V (parallela and Adaptive Co.) by 'buying' Andreas Olofsson.

This list goes on. The insurance industry is witnessing the slow disappearance of Mutual Insurance models as board vote and manipulate the conversion to corporate sector governance under the guise of ease of capital generation.

Just look deeper the list goes on. Is anyone else concerned for the open source community? There is not even the Redhat-Suse style 2 teared development and access to open source going on here.

[CAM-Gerlach]

@medmatix You'll be happy to note that unlike Rodeo (or the other major Python IDEs that I'm aware of), Spyder is 100% independently developed by a collaboratively-organized international community of volunteer and crowdfunded/grant-funded developers, and there is no one big company or monolithic organization behind it. Furthermore, there's no CLA, so the rights to the source is owned by the collective of its contributors, and so such a corporate takeover would be not be possible.

[abalter]

Hi all,

TL/DR: There are packages under active development for electron based IDEs that run both in on the desktop and web. We should build off one of those to build an IDE modeled after RStudio or combining the best of RStudio and Spyder.

I want to put a completely different idea out there. The future of IDEs is that they are based on a toolkit such as electron that shares the bulk of its code base between desktop and web. I understand that you can do this with QT as well, but electron seems to be the choice for most html/css/js based IDEs.

Currently the leaders are Atom and VSCode. VSCode has been ported to the web in the in the Coder project.

However, there is another, created by the folks who made Eclipse, that I suspect will be the most open source and likely longest maintained candidate. This is the Theia IDE. Unlike Atom it runs on web. Unlike VSCode + Coder, it 1) runs on both desktop and web natively with the same code base 2) Web and desktop are simultaneously maintained 3) It is created by a non-profit, FOSS group.

I'm game to dive in despite the fact that I'm a scientific programmer not a developer. But I can follow directions and figure things out.

Anyone out there willing to take the lead?

[abalter]

Just want to correct one thing. I just realized that Theia, although it is put forth by the Eclipse community, is based on VSCode as well. But unlike Coder desktop and web are co-maintained as I said before. I also trust the Eclipse community to keep the project going regardless of what happens to Atom, VSCode, or Coder.

[CAM-Gerlach]

While I don't want to discourage you, I want to explore your rationale a little more, as well as provide you a bit of real-world insight on the modern IDE development business.

The future of IDEs is that they are based on a toolkit such as electron that shares the bulk of its code base between desktop and web.

I've never really understood the benefits of a fully web-based IDE for serious data science work over a desktop alternative. The only one that immediately comes to my mind is being able to run code and models on a remote server (say, a cluster) from one's individual workstation, but the same thing can be accomplished without all the severe limitations of a web-based environment by simply connecting to a remote kernel on the same server with e.g. Spyder, which can be used mostly just like a local one (with greatly expanded features in this department being the main focus of Spyder 5, which is currently in the process of being funded),

However, given there clearly seems to be continued interest in this vein, I'm sure there's just something I'm missing that the web-based environment offers over the desktop for this application. @abalter , perhaps you could elaborate further on the advantages you see here?

combining the best of RStudio and Spyder.

Just to make sure everyone's clear, Rstudio (at least its frontend) is built on most of the same frameworks as Rodeo (i.e. web-based), whereas Spyder is pure Python. So in terms of code, virtually none of the latter would be translatable to this project, and you would have to accept loosing Spyder's biggest advantage over most of the rest of the field, being written in the same language as the development community it serves and thus it being much easier to solicit contributors (few scientists know JS, especially considering its such a horrible language, whereas if they're using it virtually all of them have to know Python). If you have a committed dev team or corporate sponsorship, that could be worked around but you would need to keep it in mind.

Currently the leaders are Atom and VSCode.

Currently, there is one major web-based data science IDE out there that is of paramount interest to this discussion: JupyterLab. To note, while it is usable nowadays, hundreds of thousands of $ have been poured into it just to get it ready for initial users, and its still not even a 1.0 release yet after three years of development, never mind reaching parity with desktop IDEs like Spyder in most areas and is still ultimately limited by its web-based roots (whereas Spyder has also continued to move forward despite a budget less than 1/10th of Jupyter, e.g. integrating the same LSP architecture for completion, linting, analysis, help and introspection as Atom and VSCode, among other major improvements for Spyder 4).

So, while its clearly been shown to be possible and it is difficult to compare Spyder and Jupyter directly with high confidence, the evidence and reasoning suggest creating a similar web-based IDE is likely to take far more financial resources than a desktop IDE (never mind one that can do both well), because your built-in community (Python scientific developers) by and large is much less naturally equipped to volunteer and pitch in to help, as well as the increased complexities and challenges from operating in a web-based environment.

However, on the flipside, if there isn't something your proposed IDE would provide that JupyterLab doesn't (or at least is in scope for them), it would make much more sense to just use and contribute to the development of that existing platform for your web-based needs. The one thing I can think of are the serious limitations of the whole "notebook" framework, but my understanding is that JupyterLab is not limited to just the walled garden of Jupyter notebooks and can be used to work with proper Python code files as well.

It is created by a non-profit, FOSS group.

There are basically two out there currently that take on such projects in the scientific/engineering/data science realm. Juptyer (which is almost entirely corporate-sponsored, but is itself independent) and Spyder (which is mostly crowdfunded and volunteer-developed, but has much less corporate backing). Aside from that, you'd either have to seek sponsorship from someone outside the Python ecosystem, or create your own from scratch.

[abalter]

@CAM-Gerlach

I'll try to address your points one by one:

I've never really understood the benefits of a fully web-based IDE

1. This allows you to have projects on multiple servers going at the same time.
2. If you are running code within the IDE, you can can be running code on multiple servers at the same time. 3) It is markedly simpler than connecting to a remote server. Been there done that

JupyterLab etc.

I have tried to use JupyterLab. I've promoted it at various times. Right now I'm doing biomedical data science, and I have to user R. I can use R in JupyterLab. I can even user Jupytext so that I can share the rmarkdown format between Jupyterlab and Rstudio. It's a great model. However, the way RStudio and Jupyter run the notebooks are somewhat different, so some stuff just doesn't "work.".

More importantly, even with installing various extensions and creating custom keybindings, Jupyterlab is simply not as efficient in the way I can bounce between a command terminal for the same kernel as the .R or .Rmd I'm running. I can quickly send code back and forth. Robustly inspect variables. Test things in .R files and then put them in my notebook. Quickly open files.

I didn't want it to be true, but RStudio with rmarkdown notebooks just rocks the pants off of Jupyterlab. And you can easily run code chunks with multiple kernels without needing to install other kernels or extensions.

The one thing I can think of are the serious limitations of the whole "notebook" framework

I'm not sure what to say to that. Notebooks have become the defacto way in which reproducible research is done. ALL of the work my group does is in rmarkdown notebooks. One document has text, code, images, sortable and downloadable tables, etc. I can have text that says "we found 19 significant features with log fold changes between 1.2 and 3.8" where those numbers came directly from the calculations and are put in programmatically. When we change something in the code like a filtering parameter or run a different data file, we don't need to edit those numbers by hand. Etc., etc. And that's just for our work. People are using notebooks to write books, create blog pages. It's just here to stay and only going to get better.

Aside from that, you'd either have to seek sponsorship from someone outside the Python ecosystem, or create your own from scratch.

One of the reasons I'm interested in this idea is because I'm convinced the future is beyond RStudio for R, Spyder for Python, X for Y. The future is seamless integration between the best languages for the best tasks within a single notebook.

I've put forth my idea and a super, super minimal barebones example in this repo. I don't know if I'll ever have time to develop this further. But if I did, it would be easiest in an IDE such as I'm proposing rather than using Jupyter kernels as I used for my proof-of-principle.

I don't know if I've convinced you about the utility of notebooks or the need for more flexible and future-oriented workspace environments. But at least I've laid out my thoughts for anyone else who might be interested.

[CAM-Gerlach]

This allows you to have projects on multiple servers going at the same time. If you are running code within the IDE, you can can be running code on multiple servers at the same time.

I'm not sure I really understand what you see as the major difference between these two claimed advantages, but a desktop IDE (like Spyder) is fully capable of both. You can have an arbitrary number of different kernels running on different conda envs/virtualenvs, Python installs, or remote servers connected to Spyder at any given time in one Spyder instance, and switch between them just as easily as code files in the editor (unlike Rstudio, at least when I last used it a few months ago, where you could only run one R instance at a time).

Also, if you prefer to have separate workspaces for each project, you can easily open separate Spyder instances and connect them to the same or different kernels as the first, and Spyder has a project system (that's poised to see major improvements in Spyder 4) that allows for loading and saving workspaces (files, window layouts, working dirs, your session, and soon preferences, consoles, Python environments and more, along with management of the same) that allows you to switch between them with just a click or use different projects with different instances.

It is markedly simpler than connecting to a remote server. Been there done that

If you're going to use an IDE hosted on a remote server, you still have to connect to said remote server somehow, and you still need substantial setup on the server itself to get everything working. Desktop IDEs like Spyder have built-in SSH support (and can remember your credentials and configuration settings), so all you need to do is install spyder-kernels in your desired Python environment on the server, enter your basic SSH details in Spyder, and further connections are just a click away. Not terribly difficult, and much simpler than setting up and running a whole server-based IDE I would think (at least without substantial work).

I have tried to use JupyterLab. [...]

it would be easiest in an IDE such as I'm proposing rather than using Jupyter kernels as I used for my proof-of-principle.

I share many of your reservations about JupyterLab, as I've mentioned above, and I'm certainly not arguing it doesn't have a way to go in order to come close to meeting the vision you express. However, what I'm not sure I'm getting here is how it is fundamentally incompatible with what you are trying to do, particularly considering you seem to strongly favor the whole notebook concept JupyterLab is built around, and the primary goal of your vision is build an IDE that can be both web-based and have a desktop client, preferably electron-based (and in fact, there exists an electron-based desktop client for JupyterLab. albeit one on which appears to have paused its development.

Jupyterlab is far from a perfect solution, but the fact of the matter is that reinventing the wheel as opposed to improving or adapting Jupyter (IPython) kernels and the basic infrastructure they're built on is almost always going to be the better choice, considering the alternative is somehow finding the time (over a decade), money (a least hundreds of thousands, and perhaps even millions of dollars), and people (hundreds of trained developers), just to build what they already have, much less go beyond it. The whole idea of open source is building upon others' work, and having your work built upon by others (which is, incidentally, one of the biggest shortcomings of Jupyter notebooks themselves as a do-everything tool). Given the massive investment and considerable buy-in to the Jupyter ecosystem (as you mention), as well as nearly universal adoption of Rstudio and its ecosystem in the R world, and the huge companies and organizations that back them, simply offering a better version of what already exists today, but starting from scratch instead of actually leveraging and building upon those them is simply not a viable proposition in the reality of today's world.

rmarkdown notebooks

I didn't want it to be true, but RStudio with rmarkdown notebooks just rocks the pants off of Jupyterlab.

R Markdown /= Notebooks. RStudio is trying to paint them as such to jump on the hype train of the Jupyter notebooks bandwagon, but they're really quite different (as you discuss), which ultimately is a product of the fact that "R Notebooks" are merely a pretty presentation view of R Markdown documents, which are themselves a simplified (Markdown vs. LaTeX) version of full knitr documents, which are themselves an evolution of R Sweave files.

I love R Markdown (and full-on knitr even more so, with the expressive power of LaTeX combined with the executability of actual code) and have always found it a very useful tool for what it is, because it doesn't try to be (like Jupyter notebooks) what it isn't, something more than a final knitted output format rather than an end in and of itself. Its the biggest thing I largely missed switching from R to Python, although we've been working on Spyder-Reports, which integrates pWeave (directly inspired by R Sweave, knitr and R Markdown) directly into Spyder, just like R Markdown in Rstudio (we also have Spyder-Notebook, which does the same thing for Jupyter Notebooks). They are a great tool for short papers, reports, demonstrations, exercises and more, and I look forward to fully having them in Python (and, like you, greatly prefer them to the whole Jupyter notebook model). However, what's important to understand is they aren't intended to be the primary tool in one's research toolbox, but rather the format to knit together your very highest-level, tip of the stack code and final output, statistical summary, plots and visualizations.

Notebooks have become the defacto way in which reproducible research is done.

If you think notebooks are the ultimate tool for truly "reproducible" research (much less genuinely re-usable research, in the true "shoulders of giants" spirit of both science and open source), then I suggest you read the presentation I linked. Notebooks and knitr/R Markdown certainly have their place, and can be fantastic tools for sharing and communicating the output of research, and even, in many disciplines, kitting together at a very high level underlying data processing, analysis and visualization routines. However, by design, they are pathologically unsuited for being the foundation of a truly "re-usable research" workflow, where scientists encapsulate their code into usable, sharable packages that can build upon existing frameworks and tools, be easily employed and extended for future research, and in turn serve as a robust, maintainable, interoperable foundation for those of other scientists.

I don't know if I've convinced you about the utility of notebooks

As I've said, notebooks have their place in sharing results and visualizations, and sometimes knitting together, at a high level, separate pieces of external, low-level processing and analysis code in various languages (and knitr/R Markdown is a superior tool at that job); I think we agree on both of those. My point is that they are not, by design, the be-all end-all of "reproducible research" that some of their advocates and users try to paint them as.

the need for more flexible and future-oriented workspace environments

Again, other than being web-based for the sake of being web-based, I still don't really understand the really compelling advantages of such an environment, other than being able to integrate multiple languages in one document, something which Rstudio, JupyterLab, etc. already do to varying degrees of success (which can be built-upon and improved, rather than throwing all that work out and starting over). Furthermore, as a desktop IDE, Spyder already has most of the core infrastructure in place to support the same thing; it would merely require some additional time investment to add full support for the language(s) of interest (far less work than creating a new environment from scratch). There may be a compelling case, but so far I haven't really seen it spelled out.

[david-bonin]

I think Rstudio is well on its way to becoming a fully integrated IDE for Python and R. Last year, when it came out with the built in terminal functionality, it made it easy to fire up python from within Rstudio in a way that was more natural than reticulate (to me). The beta version of Rstudio 2 is very promising and the development team has said that they want to push the python functionality even further, for instances by allowing the python console to interact with the data viewer and chart functionality like it does in Rodeo. Rstudio suits my needs very well, but I love Rodeo's neatness very much.

[stevekm]

However, what's important to understand is they aren't intended to be the primary tool in one's research toolbox, but rather the format to knit together your very highest-level, tip of the stack code and final output, statistical summary, plots and visualizations.

I strongly agree with this. Over-reliance on the "notebook" style systems is a mistake. I use R Markdown extensively, but its best usage is reporting on results that were produced externally in some controlled manner, not for actually doing the full analysis workflows themselves (example; my main production repo here uses a modular .Rmd-based system). I spent a long time with the "notebook-first" mentality described in the presentation linked by @CAM-Gerlach and the pitfalls shown there are spot-on.

I also agree with the sentiment that future efforts should be focused on supporting existing projects. I would lean more towards RStudio because in my experience, you dont need much more than Atom + terminal for general Python coding. Rstudio's strengths are in the data viewing, which is where you start to benefit the most from an interactive environment.

[medmatix]

I agree with Stephan. I have never considered Jupyter a programming development environment but a reporting environment like rmarkdown.

I think the eclipse project would be the best starting point for a full development environment for python. It is still just and afterthought plugin without the tight integration I seen for R in RStudio. This really requires a devoted group to tackle a broad project to integrate all kinds of python developer and data science needs into the Eclipse framework.

I could see where this would be a major brand forked from the core eclipse environment. Smooth integration with R an C++/Cython would be essential. I think the Walware developer has done a worthy job with StatET to bring R to Eclipse but it hasn’t ’t gone far enough for me to leave RStudio just to get python integration.

A port of RStudio to Eclipse would also be a start.

Most of the time I rely on Spyder for python and RStudio for R and Rarely when necessary Code::Blocks for C++.

David

Sent from my iPad

On May 13, 2019, at 1:41 PM, Stephen Kelly notifications@github.com wrote:

However, what's important to understand is they aren't intended to be the primary tool in one's research toolbox, but rather the format to knit together your very highest-level, tip of the stack code and final output, statistical summary, plots and visualizations.

I strongly agree with this. Over-reliance on the "notebook" style systems is a mistake. I use R Markdown extensively, but its best usage is reporting on results that were produced externally in some controlled manner, not for actually doing the full analysis workflows themselves (example; my main production repo here uses a modular .Rmd-based system). I spent a long time with the "notebook-first" mentality described in the presentation linked by @CAM-Gerlach and the pitfalls shown there are spot-on.

I also agree with the sentiment that future efforts should be focused on supporting existing projects. I would lean more towards RStudio because in my experience, you dont need much more than Atom + terminal for general Python coding. Rstudio's strengths are in the data viewing, which is where you start to benefit the most from an interactive environment.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[abalter]

@medmatix -- I think Theia IDE is the new Eclipse. It should be easier to extend as well. Also, it is based on VS Code.

[medmatix]

The current stable release of Eclipse is 4.11 (2019 03) written mostly in Java: https://www.eclipse.org/eclipseide/

Theia IDE is an unrelated open source project written in Typescript (JavaScript): https://www.theia-ide.org/ and indeed based on Microsoft’s open source source-editor Visual Studio Code (VS Code).

They are quite unrelated.

Theia is likely easier to extend but there is probably a lot of opinion to consider in such a statement. Java is less common these days as a development choice. Spyder is written in python.

Eclipse is truly polyglot with each plugin providing the bridge to a relevant interpreter of compiler.

David

Sent from my iPad

On May 13, 2019, at 3:28 PM, Ariel Balter notifications@github.com wrote:

@medmatix -- I think Theia IDE is the new Eclipse. It should be easier to extend as well. Also, it is based on VS Code.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[CAM-Gerlach]

I do note that Spyder is easy to extend with (pure-Python) plugins, and Spyder 4 will introduce a proper public API for them. In fact, behind the scenes, almost of Spyder's own features (from the Editor and the Console to the static analyzer and the profiler) are actually just "plugins" internally, and so external plugins are essentially almost as capable as internal Spyder modules.

Right now, we have a number of officially developed plugins in various stages of development (including Spyder-Notebook, adding Juptyer notebook integration; Spyder-Reports, adding R Markdown/sweave-like document creation (based on pWeave, which is based on Sweave for R); Spyder-Terminal, adding cross-platform system terminal integration; Spyder-Unittest, adding built-in support for Pytest, Unittest and nose; Spyder-Vim, adding Vim commands and shortcuts; Spyder-Autopep8, with automatic formatting; and Spyder-Line-Profiler and Spyder-Memory-Profiler, and plans for others; although they (except for Spyder-Unittest) are mostly paused right now to focus our limited resources on Spyder 4.

We always appreciate new developers to work on them, particularly Spyder-Notebook and Spyder-Reports, which are similar to what @abalter seems to be wanting.

[abalter]

@medmatix if you do a google search for theia site:eclipse.org you will find that Theia IDE is being developed by the Eclipse project, and that they are adopting it as the editor for the Eclipse Che development environment. Quoting from here,

Today, every editor or IDE project either focusses on running in browsers (e.g. Eclipse Che’s IDE) or as a local desktop app (e.g. the classic Eclipse IDE). Technologies like Electron, however, allow to run the same code in the browser as well as in an integrated desktop app. Therefore, it is no longer a choice of either desktop or cloud IDE but both scenarios can be supported with a single tool.

[fkromer]

What's the defacto tl;dr suggestion for a RStudio equivalent in Python right now? According to this R Community thread support for Python in RStudio already improved. However it was not clear to me if it's really usable e.g. in comparison with JupyterLab usability.

[abalter]

@fkromer Spyder. It's a great IDE. But also different. They have recently implemented a notebook format.

There is more information if you read upwards discussions with @CAM-Gerlach .

[OldGuyInTheClub]

What's the defacto tl;dr suggestion for a RStudio equivalent in Python right now? According to this R Community thread support for Python in RStudio already improved. However it was not clear to me if it's really usable e.g. in comparison with JupyterLab usability.

Spyder with the Notebook plugin comes the closest. They can be set to share a kernel which allows the variable inspector to show items entered in the Notebook which can be convenient. AFAIK, JupyterLab has no variable inspector (the floating widget isn't useful in my experience.) The Spyder Reports plugin while still shown on the Spyder website is no longer under development and is undocumented.