Notion "Rate limits" and Apps Script "timeout" won't allow to finish sync

astronaut954 commented 1 year ago

I'm trying to sync my Calendar with recurring dates. All the events in this calendar are recurring. There is a total of ~~45 events~~ 55 events that will recur annually, considering that each event will have a total of 30 copies ~~(so far is what the script sent)~~ there will be a total of 1650 pages. The problem is that there is this "Rate limit" that Notion has that you can find here :

Since I have to first do a FULL_SYNC to get all the events it will have A LOT of requests, so at some point, I will get this error:

I waited for the minutes it asks and ran the script again. Yet I found an issue, it will look for updates in every single event, even if the IGNORE_RECENTLY_PUSHED is turned on. I looked for the total of pages in the database and I have 1322, so I guess it is missing ~~only 28 of all the pages I need~~.

Any idea on how to solve this? I thought about setting a delay after the Rate limit has been reached, so it will wait a few minutes and after this time it will continue the script from where it stopped, yet this would also need to cancel the next time trigger, so it won't conflict.

UPDATE:

I've managed to solve the problem by deleting some events, cleaning all the database, and starting a fresh FULL_SYNC. Yet I'm facing another problem now. If I update too many events at once, I will get a TIMEOUT (30 minutes) of the Apps Script. Even if the Notion Rate Limit hasn't been reached, I get this TIMEOUT because the script has too many new entries to look at, it does get really big very easily. The script doesn't seem to continue from where it stopped, so it gets "stuck" because it will loop infinitely.

Also, I've noticed some strange behaviors now, here is the list:

New recurring events (they are all annual) now will go till 2099, so I will have 77 pages for EACH individual event.
I had to delete some duplicated events, that were created after deleting and adding the exact same event even though I didn't recover it from the trash bin, (I had to delete to do the FULL_SYNC without reaching the Rate limit, so after doing it I added again all the events by creating new ones with the exact same settings). And when deleting those duplicated events the Apps Script console would say it couldn't find the Event ID even though there was an ID in the page.
```
21 de nov. de 2022 04:02:26 Depuração   [-GCal] Deleting cancel tagged events from GCal
21 de nov. de 2022 04:02:26 Depuração   [-GCal] Error. Page missing calendar id or event ID
21 de nov. de 2022 04:02:26 Depuração   Archiving cancelled event.
```

Sometimes I will get this warning while the script is running:

21 de nov. de 2022 04:11:23 Depuração   [+ND] Event found 3mamd7j3m44vqp9tsivt3dde9u_20321226 Fim da URSS (1991) (12/25/2032 -- 12/26/2032)
21 de nov. de 2022 04:11:24 Depuração   Found multiple entries with event id 3mamd7j3m44vqp9tsivt3dde9u_20321226. This should not happen. Only considering index zero entry.
21 de nov. de 2022 04:11:24 Depuração   [+ND] Event 3mamd7j3m44vqp9tsivt3dde9u_20321226 database page 8fb23e50-748e-4997-89e7-3fcd2bf4624b exists already. Attempting update.

Sometimes I will get this strange error:

Error: <!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Notion</title>
    <style type="text/css">*{box-sizing:border-box}html{margin:0;padding:0}body{-webkit-font-smoothing:auto;align-items:center;background:#fffefc;color:#37352f;display:flex;flex-direction:column;font-family:-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,Apple Color Emoji,Arial,sans-serif,Segoe UI Emoji,Segoe UI Symbol;height:100vh;justify-content:center;justify-content:space-between;line-height:1.5;margin:0;padding:40px 20px;text-align:center}.top{height:40px}.details{align-items:flex-end;display:flex;font-size:13px;min-height:100px}.details h1{color:#999;display:inline-block;font-size:13px;margin:0 .75em}.details p{display:none}.details ul{color:#999;display:inline-block;margin:0;padding:0}.details li{display:inline-block;list-style:none;margin:0 .75em}.show-details-toggle{color:#999}.main{padding-bottom:20px}.main a{color:#37352f}a:hover{color:#e03e3e}@media screen and (max-width:960px){body{padding:20px}.details h1{margin-bottom:1em}.details h1,.details li{display:block}}</style>
    <script type="text/javascript">document.addEventListener("click",function(e){if(e.target.classList.contains("show-details-toggle")){e.preventDefault();var t=document.getElementsByClassName("show-details-toggle")[0],s=document.getElementsByClassName("details-expanded")[0];t.style.display="none",s.style.display="block"}},!1);</script>

</head>
  <body>
    <div class="top"></div>
    <div class="main">
      <img
        alt="Notion"
        style="width: 40px; margin: 0 auto; display: block;"
        src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAFAAAABUCAYAAAAVgLC7AAAACXBIWXMAABYlAAAWJQFJUiTwAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAXUSURBVHgB7ZzfVes4EMYnd/cdqABTAXSwSQVABQmP+wSpgORxn4AKEioIVEBuBUAFmAoS3vbNq8+OuLpa/xtJtiXD75w5IYkJ6MuMNSPLM6AwiJTH/Z3tCTsQdqi8F2nHg62wF2FLYT+FxeSQAbXLvmYR/RJCiiJfV49zRSxsJuyeHGEjYKQ8Rtpr8ApVJPU9H3gQNiUH3qgLiAGfUX546N4TOjE58EYpYCRsIWxIgbK/v/+bSV5eXmi73Zb96q2wOWXnSjYQEH/tmTwKsSiKPoXAz+pre3t7ue+VARFvb2/p/r7Q2WJhIzIIaQh4JeyGGkAfvBTg4OCADg8PP9+rK4QtEPL8/JziOC46ZEaZN9YGAsL7TsoOUkND9QQIIX9WRdLDyCcg3mw2c+qNSZFdX18nm80m6SM3NzeJ+JKLxr4RNq4j3o+yNweDgbeeZMvV1RU9Pz8XnTYw6CVlE2tU9VlJmQl3T/qOELNMgzeqyE6SKjs5OUne3t6SPrNYLBLhjWU6XBsLCMOH911EjA/OUuGNkSreYPdGbZBPXV5eUsjINAYJNkw+//j4oPV6TQ8PD1UfgTIQCThfQIA0QMzQ1AVy0AADV5+/v79/vl726AgIODUSEGD2enp6Yie/RYPSB697hyqUR8yNBZQMh0M6Pj5Ok2qRM6ZhAHTvcPzt+8LWWsCvzg/6xoo/qeeodblar8vnQC5s6Ks88rjlcknT6TT3870OYdvBuyxDR6NRmuLoNOaB+spMl4N3gUiwmxPw7OwsTa51L+kTRWOyFhCirVYr+qpYz8LIA78y1gI2vQzvO995oCXfAlrS20QaKQfqb9TmqMer6nTTZTovBFRXZuTA5MBPT0/THIzDxcVFWj20QWsCym/59fWVtT4HIXDxp25u+fj4yBYPEyG+KBNaERBCoRQyWdLC78zncxKXIWsdj1CtA9Kv8XicPtpkEq1MIvAKm/VAeG5eGZXHZDIhcXmDxPWNdMEXq+d5QDgca5uGteKB2FJhC85rnFBWt4w0SSseKC4ZpqEFAeAVCB0uMpR9o7U8EJ6D2RShc3R0RCZwQrktgkukixY2uyI4AXE+9SmUgyzlMLO6mJhcEGwt7EsoBysgJpO7uzvqGq8ExCzNyd0Qyl1fsPdKQKQ6yBnrgsUGJNhd4l0II0/E7tG6dB3KXp4DsfMrlFD2UkCEct3VF9BlKHs7C+NaM+eKX1eh7HUagwmFc5G+i1D2WkCcBzk7YbsIZe8TaczI3FCuscfZGUFUIpwJBcAL29oOHISAqFCKlubzgHht1crB1MJy91ddcGWujcXXYATklnmgjVAOajWGW+YhpWk6NwxuOQtpDSc3bDqMgxPQJJSbJMgFVW6Z1yTBrkhzy7ymCFZAbpnXFEFvsOSWeU0AAWMKGG6Z5xoI+JMChlvmuQYCLilwuGWeSyDgemfB0mVuKCcRLy7zo2rATlZMDljT46wuc8u8JsBfT7jG6SuDLkhoL1LRGeM3m0wmrM+vaF1i9H8DHJ/3OWoagyYKa2oAuUcabQGwQsLZGMRJUxDKdfo44DNNN5VX/g+U9UZx6oEVnYEKm/2sVqvEBPR/GY/H/+t7IyabRAicmFDkgfoeaSyenVPW0c0ZuC8Y9Wte1zegt7+zvWUWn4MFVRi8v8lbcPM2mSO+MKk4y1Bxcu/qBN90elNUyuF8aNVbtG8UrWyX1cJwmZi+SSm6VFomICQfkWFz1j6BPdlFOWnVjTYxZefDReEBgXUk0ltJ6Tc36nd1VnUBrnOn0pKyHquXZf9QGxT13ap7S2sDpK2f6lLYrLaqm1veQKqajgXSd2vNETCiTMTchEreiQRivzuuuWRCTIZkUC/31NJi4w/iEVPWLmpIXxeE1D/C/hb2LyeEVZ6oHyJu6VeaFmvP35XX5WNMWm5sKqAv/ffj3aMcuHz+sXu+UX7OM2tMBQQRZb3nx2QG99vPe+wcGwElmHr/EoabgLG80tq37wP/AffpKvnPYm+qAAAAAElFTkSuQmCC"
      />
      <p>
        Hmm... something’s not right<br />
        Please visit our
        <a target="_blank" href="https://status.notion.so/">status page</a>
        for more information.
      </p>
      <p>We’re working on it!</p>
    </div>
    <div class="details">
      <div class="details-collpased">
        <a href="#" class="show-details-toggle">Show error details</a>
      </div>
      <div style="display: none;" class="details-expanded">
        <div class="cf-error-details cf-error-504">
  <h1>Gateway time-out</h1>
  <p>The web server reported a gateway time-out error.</p>
  <ul>
    <li>Ray ID: 76d9f215ab63b17e</li>
    <li>Your IP address: 34.116.22.106</li>
    <li>Error reference number: 504</li>
    <li>Cloudflare Location: Atlanta</li>
  </ul>
</div>

      </div>
    </div>
  </body>
</html>

    at notionFetch(main:424:11)
    at getPageFromEvent(main:354:25)
    at parseEvents(main:222:27)
    at syncFromGCal(main:173:5)
    at main(main:32:5)

hk21702 commented 1 year ago

For the execution time out stuff, see the wiki for some suggestions to get around it. https://github.com/hk21702/YA-GCal-Notion-Sync-Script/wiki/FAQ-&-Common-Errors#i-got-an-execution-timeout-error

The recurring events being a bit excessive on larger time scales is unfortunately a GCal API thing that isn't easily fixed without setting a limit for all your events.

The deleting cancelled tag error is because it needs both the calendar ID and the event ID. If you're missing even just one, it will spit out that error in the console. It was likely due to a bug that I just recently fixed in #32. Update your script to the latest version and that error should be resolved.

The duplicate entries error is unfortunately a bit hard to solve. It is a known issue that sometimes crops up due to synch and timing issues with talking with the APIs.

The big error you got is Notion's issue. It appears that they were down at that time.

astronaut954 commented 1 year ago

Hi @hk21702, thank you for the response! I used the wiki suggestion (sorry, didn't see it before) and now I have a more reliable database. Limited to 3650 days, so I will have 10 events for each date. Yet, from what I understood, if it reaches the last event of the 10 events, it won't create new ones, right?

Also, kinda off-topic, I think in the near future Notion will add recurring tasks (not only for templates), I think it will make everything more smooth if you update the script with this new feature, since it won't need to create hundreds and hundreds of new events.

hk21702 commented 1 year ago

Glad to hear that things are working out better. You are correct in your understanding. The date limit is not one that shifts in real time. If you want to get events past the limit that you've set, you'd need to do a full sync due to GCal API limitations.

I know you crossed the last bit out, but just as a comment, it would really depend on how Notion implements it. Since in GCal you have the ability to modify individual events in a series of recurring events, depending on how Notion implements recurring tasks in databases, it could get really messy to try and translate between the Notion implementation and GCal.

github-actions[bot] commented 1 year ago

This issue has not had any activity for an extended period of time and is thus automatically being marked as stale.

@${author}, you can help out by closing this issue if the problem no longer exists, or adding more information.

github-actions[bot] commented 1 year ago

This issue has been closed automatically because it needs more information and has not had recent activity.

hk21702 / YA-GCal-Notion-Sync-Script

Notion "Rate limits" and Apps Script "timeout" won't allow to finish sync #27

UPDATE: