Frontend?

[ccrvlh]

Any interest making a front end for this? I tend to be a very visual person, and would really like a better alternative to flower. Would that makes sense?

[rsalmei]

Hey man!

You mean something stand-alone, using ncurses, with borders and stuff? Yeah, I think it would be very cool! Something like this?

Actually I'm very visual too, that's why clearly's output is very colorful, the colors have actual meaning. And personally I like very much the kind of "frontend" I've implemented here, inside an iPython REPL, and being able to mix general commands with clearly monitoring commands. I think it gives more power to the dev, being able to generate ad-hoc patterns on demmand, and even create debug scripts which include clearly.

But people are different, and I realize another more "graphical" frontend would appeal to many. Unfortunately I don't have any time to spare, let alone studying ncurses and actually implementing a functional frontend... But I would gladly help if someone was willing to do it!

[ccrvlh]

Actually I meant something like VueJS / React... Kind in the lines of what Flower has. We currently have an application in production and being able to have an endpoint and a webpage to look how things are going is pretty nice, but I agree with you with all of Flowers' drawbacks.

I will take a look at the code, but I rekon I could write the frontend in VueJS, and use WebSockets to keep the real time aspect. It would need an endpoint though and deciding which information would be needed.

Imagine something like this:

[rsalmei]

WOW, that's incredible! 👏 Now I get what you mean! We could leverage the data I already capture in the clearly engine, and make it available in some endpoint for you to display it! It could be something simple, using maybe flask or bottle (or the newer fastapi), where I could serve both this dynamic data and your static files! All this could be some optional feature in the clearly-server, like --frontend!

I like it man, VERY MUCH! If you really can make that happen, I can provide whathever backend endpoints you need!

[ccrvlh]

Awesome! Let me think of some structure, based what you already capture. I’ll try to make a simple prototype to see how we can e evolve this. And I’m not sure if websockets would be needed if served the pages straight from Flask/FastAPI, but i’ll dig a bit deeper! Let me know if have any ideas both visually and infrastructure wise! I also added you on that famous professional network, we’re on the same place, would be great to have a chat as well! Cheers.

[ccrvlh]

First sketch https://github.com/lowercase00/clearly-front/tree/master can run locally with yarn install and yarn serve

[rsalmei]

Hey @lowercase00, I can't believe!! I'm speechless, congrats man! 🤩 I've ran it locally, It's actually working! I don't know anything about frontends, but it is impressive for me........ Now we should just make it work humm?

Some images for the record (I use dark reader plugin):

[ccrvlh]

Great! I'm a total beginner in JS as well, but I guess this is reasonably simple, so guess it's a good start. What I though for a first prototype:

• Dashboard:

  • '# of online workers (card 1)
  • '# of offline workers (card 1)
  • success ratio (successful / failed tasks) (card 2)
  • total events processed (card 3)
  • card 4? no idea.
  • total tasks and status (in a specific time window determined in the settings, can be adjusted) - the pie chart
  • a timeseries with executed tasks and it's status - the line chart

• Tasks Table:

  • worker who processed it
  • date received
  • date started
  • date finished
  • state
  • results
  • traceback (if any)
  • args / kwargs
  • uuid
  • task name

• Workers Table:

  • worker name
  • worker status
  • worker details (pool, process, max-tasks, etc)
  • queues
  • performed tasks uuids and it's status

What do you think? We could think of a websocket connection and pass a JSON with a "type" in it, so all the parsing and usage it's the frontend's responsability. I'm not too sure how else to work on it, since we need realtime data for pretty much every kind of info... I'm not sure it would be reasonable to open several websocket connections with the backend.

I personally like Flask and FastAPI, but pretty much anyone of them would do, the good thing about FastAPI would be the documentation... but oh well, there are other ways.

Any ideas? Another thing to consider is that it would be nice to have only one service running (eg: the backend serving the static files), I really don't know what would be the best way to do that, we'll see.

About the design, this was a bad practice from my part: missing fonts, not really nice dark-theme transaction and everything, this is how it was looking during development:

[rsalmei]

Hey @lowercase00, I just had a mind blowing idea!!! Clearly since a long time is split in client and server, connected via gRPC. Well, all features clearly client provides ARE ALREADY PUBLISHED and available in the backend clearly server!! And it already has support for multiple simultaneous connections..... Sooooooo, what if there was a gRPC implementation in JS, for the web?? Well, it seems THERE IS!! 🎉

I found:

• https://grpc.io/blog/state-of-grpc-web/, which is a little old;
• https://github.com/grpc/grpc-web, which is the actual project; <<<<<<<<<<
• https://grpc.io/docs/platforms/web/quickstart/, which is the docs for the web platform! <<<<<<<<<<

Thus, it seems to me there's no work to be done in the backend, the same clearly server is able to serve text clearly clients and frontend clients with no modifications at all! If that proves to really work, we can include more information in gRPC Messages, or even more commands, to feed the graphical frontend and make it more powerful!

Regarding the frontend static content, we could serve it from any static site server, like this one in Rust, or Jekyll. And it would be simple to include one of them in the docker image and via docker-compose...

I'm even more excited now! It would be awesome to concentrate all server logic in one place, clearly server for the win! What do you think?

[ccrvlh]

I just pushed the base structure at a new repo (https://github.com/clearly00/frontend). I generated the typescript necessary to consume the server, using the .proto from the source code. But I have to say I have no idea how gRPC works, so I still have to study how to pull info from the server and consume it on the application. I'm guessing it shouldn't be too difficult... I'll get back with some news

[ccrvlh]

@rsalmei been a while, but I wanted to get back to this, slowly but still... I restarted the frontend project, somethings that I was wondering if you think it could make sense.

I've been using Flower daily for a while, and it's interesting to save the information and than rotate (info is normally short lived). I saw a project called Leek that uses ES, but it's way too heavy for the task.

I though of having two options: saving the stream to disk (TinyDB) or saving to SQLite Database. And then have a setting to rotate the database (delete records older than X days).

From what I've seen, the project would probably need a new layer to save the data (and not just stream to the terminal), and probably a small API for the frontend to interact with.

I could help with both, but I was wondering what are your thoughts on that. If you find this could make sense, would be great to have some sort of CONTRIBUTING.md explaining a bit more the project structure and what practices to follow when making PRs.

Cheers!

Edit: Just forked the project, I'm thinking of having a new api module (FastAPI powered) with the routes, and then a db module, the user could have the (default) option to use SQLite or Postgres, using SQLAlchemy this shouldn't be a problem.

In order to save events, I guess the ClearlyClient could be used, I don't really know how (just opened the project), but it may be the case there's some serializing to do on the client? Since I guess as of now it's suited for the terminal (data format, colors and etc). Maybe passing a "serialized" parameter to the client so that it spits JSON or something instead of pretty printing text.

Let me know your thoughts on the design/implementation, I would be happy to work on this.

[ccrvlh]

BTW, any instructions on how to run w/o Docker?

[rsalmei]

Yes, they're on the readme. Look for the collapsible section "What, you really need to use it inside your own REPL?".

[adambirds]

Just wanna say, this looks epic @lowercase00. Hoping it gets released soon.