ns throws Value Error

[Acidham]

Describe the bug I am receiving ValueError when running Notes search with ns

[09:43:29.841] ERROR: Search Notes[Script Filter] Code 1: Traceback (most recent call last):
  File "/Users/aci/Dropbox/Alfred/Alfred.alfredpreferences/workflows/user.workflow.027998C4-05B8-4F90-9F44-8C2461B79E36/get_notes.py", line 126, in <module>
    print(getNotes(searchBodies=False))
  File "/Users/aci/Dropbox/Alfred/Alfred.alfredpreferences/workflows/user.workflow.027998C4-05B8-4F90-9F44-8C2461B79E36/get_notes.py", line 95, in getNotes
    folderName = folderNames[folderCodes.index(d[1])]
ValueError: tuple.index(x): x not in tuple

To Reproduce Steps to reproduce the behavior:

1. type ns
2. Fallback Search > nothig found

Desktop (please complete the following information):

• OS: 10.15.4
• Alfred version: 4.0.9
• Workflow version: 2.0.0

[sballin]

Weird! Another issue related to the Notes database. I can't replicate this for myself, but if you go to the workflow directory and do python3 get_notes.py, you should be able to run it. I recommend getting rid of the print statement at the last line, and adding the following under line 92, right before the loop:

print(folderCodes)
print([d[1] for d in dbItems])

I'd be interested to see the results. Also, are you using iCloud sync?

[Acidham]

Strange, I tested the WF on my second Mac (MBP 13) and it worked. I thought its related to wrong python version but both mac shows for python3 --version 3.8.0. The Mac (MBP 16) is the mac where the error happens.

I added your lines but it just prints list of integers on both computers. On MBP16 it shows the list of integers with the same error message.

[sballin]

On the computer with the error, can you see which numbers are in [d[1] for d in dbItems] but not in folderCodes? One way to do this:

set([d[1] for d in dbItems])-set(folderCodes)

[Acidham]

I added following lines and exec on shell:

diff = set([d[1] for d in dbItems]) - set(folderCodes)
print(diff)

On working mac: it returns series of number plus set set():

set()

on mac where it is not working it returns {None} plus the ValueError that I posted in first post.

[sballin]

Interesting, some notes have None as their folder code instead of an integer, which I didn't think could happen. Can you see which ones those are with

[d[0] for d in dbItems if d[1] == None]

and let me know if there's anything unusual about them.

[Acidham]

I am getting one Note with "Bad Entwurf Nr. 2" and I tried to search in Notes but was not able to find a note. Can it be that Notes is corrupt on the one machine? I had 2 crashes with Notes today?

[sballin]

It's possible. Did the crashes happen when you were using the workflow?

[Acidham]

No it happened during move operations, but not the note with null.

To improve resilience I would ignore The None result, log an error and proceed with exec.

I will remove and add iCloud Notes on my computer...

[Acidham]

I did a reset (remove and add from iCloud) but error is still the same.

I added following code, replacing line number 101 and 102 to get Search Notes WF back to work:

try:
    folderName = folderNames[folderCodes.index(d[1])]
except ValueError:
    folderName = None
    pass
if folderName is None or folderName == 'Recently Deleted':

[sballin]

Nice work! If you have a chance, could you test this slightly different version that I'm considering for the next release (on the computer with the original error)?

#!/usr/bin/env python3
import sqlite3
import zlib
import re
import os
import json

def extractNoteBody(data):
    # Decompress
    try:
        data = zlib.decompress(data, 16+zlib.MAX_WBITS).split(b'\x1a\x10', 1)[0]
    except zlib.error as e:
        return 'Encrypted note'
    # Find magic hex and remove it 
    # Source: https://github.com/threeplanetssoftware/apple_cloud_notes_parser
    index = data.index(b'\x08\x00\x10\x00\x1a')
    index = data.index(b'\x12', index) # starting from index found previously
    # Read from the next byte after magic index
    data = data[index+1:]
    # Convert from bytes object to string
    text = data.decode('utf-8', errors='ignore')
    # Remove title
    lines = text.split('\n')
    if len(lines) > 1:
        return '\n'.join(lines[1:])
    else:
        return ''

def fixStringEnds(text):
    """
    Shortening the note body for a one-line preview can chop two-byte unicode
    characters in half. This method fixes that.
    """
    # This method can chop off the last character of a short note, so add a dummy
    text = text + '.'
    # Source: https://stackoverflow.com/a/30487177
    pos = len(text) - 1
    while pos > -1 and ord(text[pos]) & 0xC0 == 0x80:
        # Character at pos is a continuation byte (bit 7 set, bit 6 not)
        pos -= 1
    return text[:pos]

def readDatabase():
    # Open notes database read-only 
    home = os.path.expanduser('~')
    db = home + '/Library/Group Containers/group.com.apple.notes/NoteStore.sqlite'
    conn = sqlite3.connect('file:' + db + '?mode=ro', uri=True)
    c = conn.cursor()

    # Get uuid string required in x-coredata URL
    c.execute('SELECT z_uuid FROM z_metadata')
    uuid = str(c.fetchone()[0])

    # Get note rows
    c.execute("""SELECT c.ztitle1,            -- note title (str)
                        c.zfolder,            -- folder code (int)
                        c.zmodificationdate1, -- modification date (float)
                        c.z_pk,               -- note id for x-coredata URL (int)
                        n.zdata               -- note body text (str)
                 FROM ziccloudsyncingobject AS c
                 INNER JOIN zicnotedata AS n
                 ON c.znotedata = n.z_pk -- note id (int) distinct from x-coredata one
                 WHERE c.ztitle1 IS NOT NULL AND 
                       c.zfolder IS NOT NULL AND            -- fix issues/21
                       c.zmodificationdate1 IS NOT NULL AND -- fix issues/20
                       c.z_pk IS NOT NULL AND
                       n.zdata IS NOT NULL AND              -- fix issues/3
                       c.zmarkedfordeletion IS NOT 1""")
    dbItems = c.fetchall()

    # Get folder rows
    c.execute("""SELECT z_pk,   -- folder code
                        ztitle2 -- folder name
                 FROM ziccloudsyncingobject
                 WHERE ztitle2 IS NOT NULL AND 
                       zmarkedfordeletion IS NOT 1""")
    folders = {code: name for code, name in c.fetchall()}

    conn.close()
    return uuid, dbItems, folders

def getNotes(searchBodies=False):
    # Custom icons to look for in folder names
    icons = ['📓', '📕', '📗', '📘', '📙']

    # Read Notes database and get contents
    uuid, dbItems, folders = readDatabase()

    # Sort matches by title or modification date (read Alfred environment variable)
    if os.getenv('sortByDate') == '1':
        sortId = 2
        sortInReverse = True
    else:
        sortId = 0
        sortInReverse = False
    dbItems = sorted(dbItems, key=lambda d: d[sortId], reverse=sortInReverse)

    # Alfred results: title = note title, arg = id to pass on, subtitle = folder name, 
    # match = note contents from gzipped database entries after stripping footers.
    items = [{} for d in dbItems]
    for i, d in enumerate(dbItems):
        folderName = folders[d[1]]
        if folderName == 'Recently Deleted':
            continue
        title = d[0]
        body = extractNoteBody(d[4])
        # Replace any number of \ns with a single space for note body preview
        bodyPreview = ' '.join(body[:100].replace('\n', ' ').split())
        subtitle = folderName + ' | ' + bodyPreview
        if searchBodies:
            match = u'{} {} {}'.format(folderName, title, body)
        else:
            match = u'{} {}'.format(folderName, title)
        # Custom icons for folder names that start with corresponding emoji
        if any(x in folderName[:2] for x in icons):
            iconText = folderName[:2]#.encode('raw_unicode_escape')
            icon = {'type': 'image', 'path': 'icons/' + folderName[0] + '.png'}
            subtitle = subtitle[2:]
        else:
            icon = {'type': 'default'}
        subtitle = fixStringEnds(subtitle)
        items[i] = {'title': title,
                    'subtitle': subtitle,
                    'arg': 'x-coredata://' + uuid + '/ICNote/p' + str(d[3]),
                    'match': match,
                    'icon': icon}

    return json.dumps({'items': items}, ensure_ascii=True)

if __name__ == '__main__':
    print(getNotes(searchBodies=False))

[Acidham]

I can confirm, the code above works on my "error prone" machine!

[sballin]

Should be fixed as of version 2.1.0.