Bounty Proposal: Logging Improvements

[0x4007]

Describe the background or context

Logging can help us diagnose issues more effectively.

Describe the solution

• [x] #592
• [x] #593

[0x4007]

@0xcodercrane can you please set the time label and approve or adjust this proposal?

[b4s36t4]

Right now we're using mezmo for logging service but how do we do that with Cloudflare logs? Means cloudflare is providing three different ways which one we're going with? LogPush or Log Engine?

[0xcodercrane]

If you want to get assigned to the issue, please make a detailed plan and post it here @b4s36t4

[b4s36t4]

@0xcodercrane I wanted to understand which one we're going with LogPush Or LogEngine so that I could spring out my proposal here.

[0xcodercrane]

I think instant logs are fine

[devpanther]

I think instant logs are fine

I don't think this persists the logs.. and integrating cloudflare logs into this application is gonna be a hassle.

It handles logs from websites linked to your account, doesn't really accept a HTTP logs push, retention is bad and it requires business or enterprise account

[devpanther]

Since we have a supabase, we can create a logger handler to persist and delete stale logs (if needed) and the duration is under our control, we don't have to pay if we do not have to many logs.

We do not perform read requests, just writes and we have full control to send and persist what we like.. at this point we can easily create something to consume logs from the DB and make it more easier to read and track down problems

Retention is key

[devpanther]

@0xcodercrane

[kamaalsultan]

/start

[ubiquibot[bot]]

Deadline	Thu, 03 Aug 2023 00:06:42 GMT
Registered Wallet	Please set your wallet address to use `/wallet 0x4FDE...BA18`
Payment Multiplier	1.00
Multiplier Reason	undefined
Total Bounty	600 USD

Tips:

• Use /wallet 0x4FDE...BA18 if you want to update your registered payment wallet address @user.
• Be sure to open a draft pull request as soon as possible to communicate updates on your progress.
• Be sure to provide timely updates to us when requested, or you will be automatically unassigned from the bounty.

[kamaalsultan]

/wallet 0xeFeF11577Db8f882F6Bf92F46008170Ba3602D40

[ubiquibot[bot]]

Updated the wallet address for @ByteBallet successfully! Your new address: 0xeFeF11577Db8f882F6Bf92F46008170Ba3602D40

[devpanther]

@ByteBallet This is still a proposal, you are supposed to propose a solution not assign yourself

[0xcodercrane]

The original spec was to move to Cloudflare logs. but if you think the supabase is better, would you give us pros/cons?

[kamaalsultan]

Yes, of course First of all, Supabase is based on PostgreSQL and Cloudflare is based on SQLite Supabase provides higher security than Cloudflare by supporting TLS encryption while Cloudflare doesn't have encryption Also supabase provides stronger monitoring and analytics features like log retention and statistics Cloudflare provides replication and partitioning Cloudflare automatically backups the data to R2 Supabase is mostly used for North America and Cloudflare is widely used in Europe And finally, I am much more familiar with Supabase

[whilefoo]

Just my 2 cents: Cloudflare Logs doesn't even support pushing your own logs to their service but can only export logs generated by their services such as Workers, Pages, DNS, HTTP requests... to an external service.

I wouldn't use a database for logs because it doesn't offer indexing out of the box and when you want to view logs you either have to use Supabase dashboard which doesn't offer much filtering and searching. You would have to write SQL queries for more complex logs searching or create your own UI over it.

I think something like Google Cloud Logging is really good because it offers indexing and an logging explorer that offers searching, filtering, sorting... and has its own query language but it's really simple and intuitive to use through the UI. Their searching algorithm is really fast even if you have a lot of logs. It's cheap too.

[0xcodercrane]

I am leaning toward @whilefoo 's solution because the basic features of a monitoring system are search, filter, sort, etc. we still need to consider if its worth paying right now, Logdna is also good but under free plan, we can only see real-time logs and couldn't invite members to share them.

[kamaalsultan]

I looked into Google Cloud Log and it seemed better.

[0x4007]

I just dislike the idea of breaking apart our tech stack across more services (more credentials, and more cognitive overhead for documentation around different services)

If complex queries are the hangup for using supabase, why not just generate a few useful queries using ChatGPT?

For example, we have a log view webapp that loads the data from supabase and allows for filtering?

[0x4007]

I just removed the bounty for now until we figure out where to store the logging data. I'm really leaning towards using existing infrastructure (we currently do use Cloudflare, Supabase, and GitHub.)

I understand that they might not be the absolute best tools for the job but I still believe that it will make things easier for large (distributed) team collaboration to consolidate providers.

[kamaalsultan]

If the only issues for supabase is complexity of SQL queries, I don't think it's problem. Generating sql queries is not so difficult if logic is clear.

[whilefoo]

If using existing infra is a must then storing logs in the database and creating a web app to view them is the best way. Even better if such open source solution already exists.

[kamaalsultan]

I surfed on some open source projects but couldn't find perfect match. I think the best way is to start with Supabase right now. Here, what I am not sure is "3. compare what happened on the GitHub to the logs". Isn't is same with "2. backtracing"?

[0x4007]

Yes seems that I was redundant about 2 and 3!

So it seems that the consensus is to go with Supabase. I'm okay with this for now!

However I want to propose a last, but perhaps naive idea: what if we use Cloudflare KV for logs? It's just a key value pair storage mechanism which seems pretty simple to work with conceptually. Maybe with a Unix time stamp as the key and then we dump any logs at that time stamp in the value encoded in an array? It would also be handy to post a link back to the comment on GitHub that the log is associated with (or better if we could generate the URL based on the comment "node id" etc)

[ubiquibot[bot]]

Do you have any updates @ByteBallet? If you would like to release the bounty back to the DevPool, please comment /unassign Last activity time: Tue Aug 01 2023 02:45:46 GMT+0000 (Coordinated Universal Time)

[kamaalsultan]

Yes seems that I was redundant about 2 and 3!

So it seems that the consensus is to go with Supabase. I'm okay with this for now!

However I want to propose a last, but perhaps naive idea: what if we use Cloudflare KV for logs? It's just a key value pair storage mechanism which seems pretty simple to work with conceptually. Maybe with a Unix time stamp as the key and then we dump any logs at that time stamp in the value encoded in an array? It would also be handy to post a link back to the comment on GitHub that the log is associated with (or better if we could generate the URL based on the comment "node id" etc)

I will take a look on it. I once used Cloudflare to save timestamp of uploading files to S3 bucket. But as I know, Cloudflare KV has limitations on the number of requests and storage size.

[whilefoo]

I don't think using Cloudflare KV makes sense. It's very limited in its design - we can't do filtering, searching, sorting or any sort of complex operations. It seems like you can't even fetch many keys (logs) in one API call but have to do it for every timestamp which means we can't list logs efficiently.

I don't see any real advantage using KV, because we can reference the original comment with Supabase too.

[ubiquibot[bot]]

Do you have any updates @ByteBallet? If you would like to release the bounty back to the DevPool, please comment /unassign Last activity time: Sat Aug 05 2023 05:42:59 GMT+0000 (Coordinated Universal Time)

[0x4007]

I can't easily turn the checklist into sub tasks from the GitHub iOS app so I will do that shortly from my computer.

[kamaalsultan]

/unassign

[ubiquibot[bot]]

You have been unassigned from the bounty @ByteBallet