conversion fails from the command line with errors

discdiver commented 4 years ago

Thank you for making this. Trying it out.

Just installed. Restarted JupyterLab server. I don't see the deploy option there. Figuring maybe this is only for Jupyter notebook?

Tried to run from the command line. Got the following stack trace. Any advice?

(base) Jeffs-MBP:sidetable jeffhale$ jupyter_to_medium  "sidetable7.ipynb"

[0625/142412.662474:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142412.663990:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142412.665101:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142413.483731:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmp_in42qxf/temp.png.
[0625/142413.788236:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142413.789905:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142413.791006:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142414.594881:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmpf8vrezlf/temp.png.
[0625/142414.970822:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142414.972396:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142414.973333:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142415.784465:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmpgckoi_1n/temp.png.
[0625/142416.143842:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142416.145380:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142416.146453:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142416.953669:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmpoj7egng4/temp.png.
[0625/142417.237184:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142417.238836:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142417.239976:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142418.044612:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmpexi1pfij/temp.png.
[0625/142418.325582:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142418.327152:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142418.328120:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142419.132134:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmpnlh_129g/temp.png.
[0625/142419.413139:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142419.414611:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142419.415598:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142420.240582:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmpykixj4c8/temp.png.
[0625/142420.662586:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142420.664422:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142420.665423:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142421.480219:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmprrym6tlw/temp.png.
[0625/142421.762085:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142421.763545:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142421.764376:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142422.580194:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmpiud9djty/temp.png.
[0625/142422.842650:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142422.844171:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142422.845282:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142423.654075:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmp9hp072qi/temp.png.
[0625/142423.933863:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142423.935466:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142423.936393:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142424.746912:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmp7u_6m_06/temp.png.
[0625/142425.024594:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142425.026150:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142425.027032:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142425.834788:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmpg4jbuecw/temp.png.
[0625/142426.117763:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142426.119271:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142426.120401:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142426.931917:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmpa2ni6b1d/temp.png.
[0625/142427.212889:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142427.214306:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142427.215168:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142428.032410:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmpg11qx19h/temp.png.
[0625/142428.371322:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142428.372989:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142428.373834:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142429.176188:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmprd6ugs6v/temp.png.
[0625/142429.521623:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142429.523225:ERROR:file_io.cc(89)] ReadExactly: expected 8, observed 0
[0625/142429.524005:ERROR:xattr.cc(63)] setxattr org.chromium.crashpad.database.initialized on file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/: Operation not permitted (1)
[0625/142430.329871:INFO:headless_shell.cc(620)] Written to file /var/folders/bv/7sjpwlf93qzcny33rxrqz1zr0000gn/T/tmpul7b9d17/temp.png.
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
Traceback (most recent call last):
  File "/Users/jeffhale/miniconda3/lib/python3.7/site-packages/jupyter_to_medium/_publish_to_medium.py", line 151, in load_images_to_medium
    new_url = req_json['data']['url']
KeyError: 'data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jeffhale/miniconda3/bin/jupyter_to_medium", line 8, in <module>
    sys.exit(main())
  File "/Users/jeffhale/miniconda3/lib/python3.7/site-packages/jupyter_to_medium/_command_line.py", line 129, in main
    publish(**args)
  File "/Users/jeffhale/miniconda3/lib/python3.7/site-packages/jupyter_to_medium/_publish_to_medium.py", line 289, in publish
    p.main()
  File "/Users/jeffhale/miniconda3/lib/python3.7/site-packages/jupyter_to_medium/_publish_to_medium.py", line 217, in main
    self.load_images_to_medium()
  File "/Users/jeffhale/miniconda3/lib/python3.7/site-packages/jupyter_to_medium/_publish_to_medium.py", line 153, in load_images_to_medium
    raise ValueError('Problem loading images: ' + r.text)
ValueError: Problem loading images: {"errors":[{"message":"The media is not in a format we understand.","code":-1}]}

tdpetrou commented 4 years ago

Thanks for testing it out!

The error at the very bottom is from Medium, stating that the "media is not in a format we understand". Looks like it loaded 17 images and then failed. The image must be png, jpeg, tiff, or gif if I recall correctly.

Maybe theres an image in your markdown thats not one of these formats?

Also, I just updated this package a few minutes ago, though it won't help with this error (I believe), but you might want to update regardless.

tdpetrou commented 4 years ago

Also, can you upload the notebook here (attach it in a post).

And to answer your other question - this is just for the classic notebook. Jupyter Lab seems to have a completely different interface for adding extensions, though I haven't looked deeply, and if someone knew what they were doing, I'd imagine it wouldn't take too long.

discdiver commented 4 years ago

Thanks.

Images are .jpg or .png. There are some emoji, but those should all be unicode characters. No image types I haven't used on Medium before.

I updated the package. As you expected, same result.

I tried switching from chrome to matplotlib for table conversions, in case that was the issue. Same result.

It could be helpful if the error message told me which image failed. Might be something worth adding to future releases.

discdiver commented 4 years ago

Notebooks can't be attached to GitHub issues comments, apparently. Learn something every day. I'll link to a GitHub repo.

discdiver commented 4 years ago

GitHub should have sent you an invite to collaborate for the repo, @tdpetrou

discdiver commented 4 years ago

Re: JupyterLab, I wish I were a person who knew building extensions, but I don't yet.

Might be worth saying explicitly in the ReadMe that this is just for classic Jupyter Notebooks so folks know that.

discdiver commented 4 years ago

jupter_to_medium appears to be trying to upload a bunch of images when it fails, but only the 4 files I created are in the ..._files folder. Images generated from the DataFrame html ouptput tables are not in that folder. I don't know if they are supposed to be there?

tdpetrou commented 4 years ago

Attempting to answer all q's here:

Yes, a better error message is something to add
I think you can zip the notebook and add it here
I'll join the collaborate (hadn't heard of this before)
Agreed, will update readme to mention only classic notebook (will work on jupyter lab when i have time)
DataFrame images will not be in ..._files folder since they are saved bytes

Can you test out a really short notebook with maybe one markdown image and one dataframe?

EDIT - just downloaded the notebook and it failed for me a well.

discdiver commented 4 years ago

I was able to upload a few short notebooks with all the same images I added to the other notebook. 🎉

Then I got a different error when I ran the script to upload after I added a sidetable method. I'll make a new issue for that as it appears unrelated to this one. Although, I wonder why I didn't get that error with my previous notebook. 🤔

tdpetrou commented 4 years ago

I think I see two problems. One, I need to change the extension to 'jpeg' if its 'jpg' and two, the _files directory must be deleted no matter what before the start. This is why you are getting different errors. the _files directory still contains old images.

In _.publish_to_medium.load_images_to_medium add this line

if extension == 'jpg':
    extension = 'jpeg'

and i think things will work. Make sure to delete the _files directory. I'll have a fix for this soon.

tdpetrou commented 4 years ago

Hold up. Making that change doesn't work, but changing the image extensions themselves to jpeg does. I'll have to make the change to jpeg earlier.

discdiver commented 4 years ago

Hmm. Maybe I'm misunderstanding, but I tried deleting the files directory earlier several times to no avail.

For my short test cases with different issues I didn't delete the images folder and I repeatedly ran the script and it uploaded everything fine.

Also, when I looked at Medium's guide to images, it said it would take .jpg and .jpeg:

Note: The editor supports images up to 25MB in the .JPG, .JPEG, .GIF, and .PNG formats. https://help.medium.com/hc/en-us/articles/215679797-Images

Maybe a dumb question, but where do I access _.publish_to_medium.load_images_to_medium? I installed in a conda environment using pip.

discdiver commented 4 years ago

The images upload as .jpg just fine in a brief example notebook.

tdpetrou commented 4 years ago

That linked article is for posting not API. The API only uses JPEG - https://github.com/Medium/medium-api-docs

I've made the change in another place and am able to get the entire notebook to load

I'll have a fix for this soon and remove the creating of the _files directory unless explicitly saving the markdown.

discdiver commented 4 years ago

I made some changes to the notebook, deleted the image folder created by jtm. Some images seemt to load ok. Then I get a JSONDecodeError when trying to do table_conversion with matplotlib

loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
loading image to medium
Traceback (most recent call last):
  File "jtm_script.py", line 15, in <module>
    table_conversion="matplotlib",
  File "/Users/jeffhale/miniconda3/lib/python3.7/site-packages/jupyter_to_medium/_publish_to_medium.py", line 311, in publish
    p.main()
  File "/Users/jeffhale/miniconda3/lib/python3.7/site-packages/jupyter_to_medium/_publish_to_medium.py", line 231, in main
    self.load_images_to_medium()
  File "/Users/jeffhale/miniconda3/lib/python3.7/site-packages/jupyter_to_medium/_publish_to_medium.py", line 163, in load_images_to_medium
    req_json = r.json()
  File "/Users/jeffhale/miniconda3/lib/python3.7/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/Users/jeffhale/miniconda3/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/Users/jeffhale/miniconda3/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/jeffhale/miniconda3/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

discdiver commented 4 years ago

Gotcha re: API jpg vs. jpeg. But that's strange because I included the same .jpg files just fine in a sample notebook and everything uploaded fine. I didn't rename them to .jpeg

tdpetrou commented 4 years ago

Just released a new version with the updates. No files will be created locally, unless save markdown is chosen. jpg are sent to Medium as jpeg.

Can you upgrade to 0.1.1 and try again?

discdiver commented 4 years ago

Thank you, Ted! Updated to 0.1.1 All good.

dexplo / jupyter_to_medium

conversion fails from the command line with errors #5