search

vsoch / freegenes

FreeGenes BioNode with Django
https://vsoch.github.io/freegenes/
Mozilla Public License 2.0
2 stars 4 forks source link

Shipment information #14

Closed Koeng101 closed 5 years ago

Koeng101 commented 5 years ago

Right now we are mostly using the Shippo API to handle shipments. The reason for keeping data about where we are shipping and such is in case we want to move to a different API or handle shipping through idiosyncratic means.

@vsoch what is your recommendation on how to implement shipping?

vsoch commented 5 years ago

I don't have a concrete recommendation (never done it!) but my intuition is that we need to stay away from storing personal information to the best that we can. Can you point me to what you are currently doing so I can take a look? (And I found the Shippo API docs so I'm good there :+1: )

vsoch commented 5 years ago

If you don't have code/documentation, a simple explanation works too :) Something like:

This actually might fit well into a user story if you just want to add it with those to send along.

vsoch commented 5 years ago

And to be clear, we definitely want to automate this maximally, and this particular user story might not match the "in the future ideal" but it will be a good start to get my head where the lab is currently operating.

Koeng101 commented 5 years ago

If you check out routes + models for shipment stuff in our current API, you should find the code-process we run right now. Basically:

  1. User makes order
    • Includes their address and what distributions they want
  2. OpenMTA is signed idiosyncratically, and then uploaded to the API (the order then points to that MTA)
  3. We create shipment from order
    • In addition to their address in the order, we also add in our "from" address and the type of parcel we're sending, including some information necessary for Fedex / customs
  4. From the shipment, we can retrieve rates. We choose a rate that looks appropriate for what we want to ship and confirm we want that particular rate
  5. From "/shipment/retrieve_transaction/{uuid}" we get the shipping label, which we just print out.
  6. Fedex picks up the box

My thoughts on storing shipment information: If a malicious actor gets access to the running server, they'll be able to get addresses / personal information anyway. If a malicious actor gets access to our database, we're still kind of screwed. However, storing the addresses locally allows us to use different APIs eventually, which is why I went with it - but there might be a different way that is better!

vsoch commented 5 years ago

If a malicious actor gets access to the running server, they'll be able to get addresses / personal information anyway.

Not if there isn't any stored on the server or database, they'd maximally get usernames and email addresses associated.

If a malicious actor gets access to our database, we're still kind of screwed.

Why? I mean yes it would be terrible, but from a personal information standpoint, if there aren't people's addresses, names, shipping details, they won't get that.

However, storing the addresses locally allows us to use different APIs eventually, which is why I went with it - but there might be a different way that is better!

Here are some ideas I'm brainstorming now:

  1. The most basic thing we can do is implement shippo (in the same way you have) directly with the application, and have the shipping information be ephemeral - meaning that the user enters it at the time of the order, we use it to generate labels (I'm not clear what this actually looks like, I'd want to test) and then it gets thrown away.
  2. We can improve upon that by storing variables in the client side (local storage or similar)
  3. We could have a separate Google Sheet, indexed by user email (unique in the database) that will send a user to a Google form to populate if the email isn't found, otherwise populate from the sheet. This would mean using the sheets API from the server, and in the case it's accessed, it would be an extra step to query the API, and the credential could be revoked.

I'm leaning toward (to start) trying out the local cache method, and we'd have to see how it performs with different browsers. That would be the sure fire way to not store any personal information anywhere, but still have it easy to do for the user (and not have to re-enter).

A few more questions about your comments:

I think (before we dive into implementing the shipping bit) we need to do the views - meaning I create the page where a user creates an order (whatever that looks like) and then think through what happens when they click submit. To drive the views, I'd really like to have an export of the current database - would that be possible? That should give me a good chunk of work (to write the migration scripts) while you are in Boston. Let me know what you think!

Koeng101 commented 5 years ago

Not if there isn't any stored on the server or database, they'd maximally get usernames and email addresses associated.

If they get our Shippo API token, which would probably be a local variable, they can just call "https://api.goshippo.com/addresses/" to get all the addresses

Why? I mean yes it would be terrible, but from a personal information standpoint, if there aren't people's addresses, names, shipping details, they won't get that. That's true, they wouldn't get personal info so long as they didn't get the token (assuming we use shippo)

We would still want to track orders and shipments, but could probably just store the Shippo object IDs instead of all the information we currently store. Unless a malicious actor gets the token, we'd be fine then.

  1. Yes
  2. They include their address in the MTA that their institution signs
  3. The MTA is signed by the recipient sending their TTO the MTA, usually as an email attachment. This seems really annoying, but TTO's really don't adopt tech quickly, and quite a few don't use online signing services. It is then sent back to us
  4. I create shipments manually (which is why I'd like that to be automated, spent about an hour or two yesterday just filling out and sending post requests to our API)
  5. "/shipment/rates/{uuid}" on our API gets us a variety of rates. I usually just pick out the fedex standard overnight rate.
  6. It does not, just requires saving the shipment object id

Sure, will get you that over email.

vsoch commented 5 years ago

ah ok, I didn't realize the addresses were available via the shippo api (I thought it was just used, one off, to generate a label and then the information went away).

I think having the token (and then storing order IDs) is a good start, it's not perfect, but minimally the token can be revoked.

Koeng101 commented 5 years ago

One thing to note is that personal information + shipping information is stored within the material transfer agreements that we have to keep for each order. They're PDFs, which makes programmatically ripping them out harder, but a malicious actor probably could still access them

vsoch commented 5 years ago

okay thanks for letting me know. Is it primarily names and addresses, is there anything else?

vsoch commented 5 years ago

Either way, data won't be exposed (available as a URL) from the server https://github.com/vsoch/freegenes/commit/e9314d0d8539f6ad575dd2fc61829e2a55d0490b

Koeng101 commented 5 years ago

Sent email with example of MTA.

hverdonk commented 5 years ago

I've collected a few common questions our users have sent so far.

  1. Is there any info on the sequences of the parts & the backbone they're in?
  2. Are parts sent in cells, or just as raw DNA?
  3. Is there something online to tell me what parts you sent? (I take this to mean "What is included in, say, the OpenEnzyme collection?")
  4. Do you need me to fill out the survey each time for each plate? (explanation: we have a QR code on a packing slip that links to this google form.
  5. Several people requested their FedEx tracking numbers so that they could track their package.
  6. What are our collections? (This is the biggest question.)
vsoch commented 5 years ago

Can you tell me what a backbone is, in context of sequences / parts?

For question 2 - this is something that can be answered in an FAQ-like page. For example, here are the current FAQ, and that could be linked from an order confirmation / notification, any kind of communication. Actually, I think we should migrate the FAQ to the main docs site (maybe under getting started) so it can be better collaborated on (and we don't need to update the server to update the content).

For question 3,5, this would be logical to have provided via the Order / Shipment page, which would be viewable by the individual that generated the order and then FreeGenes staff/admin

For question #6, this would be easy to answer with a search interface. I've implemented basic search (across everything) and am going to do a "Category specific" search where a user can drill down (e.g., Collections -> something else, etc.).

vsoch commented 5 years ago

If I create an FAQ skeleton page for the docs, would you like to contribute some of the additional questions that you have in mind? We could organize it based on the different interactions (e.g., shipments, inventory) and then I'll move it off of the about page.

hverdonk commented 5 years ago

Can you tell me what a backbone is, in context of sequences / parts?

It's the plasmid vector. People want to know it's sequence.

For question 2 - this is something that can be answered in an FAQ-like page.

2 and 4 are best answered in an FAQ page, I agree. I'd be happy to contribute to the page as well.

For question 3,5, this would be logical to have provided via the Order / Shipment page, which would be viewable by the individual that generated the order and then FreeGenes staff/admin

5 should be. I believe that 3 was referring to what parts were in each collection, rather than what plates are in each shipment, which might be better answered elsewhere (as there are 96+ parts per collection). It might be best to do this with the search interface, as people would likely search for a specific part, then order it's collection.

For question #6, this would be easy to answer with a search interface. I've implemented basic search (across everything) and am going to do a "Category specific" search where a user can drill down (e.g., Collections -> something else, etc.).

This would be perfect!

vsoch commented 5 years ago

thanks for the clarification - if you are referring to the "vector" field of a part, if a user creates an order, and there is an order detail page, the order part that they ordered should link to it's detail page with the vector.

I'll pause on search / detail views and get you an FAQ page to edit asap!

vsoch commented 5 years ago

okay - just added a faq page and you can edit the file here! I did a best shot guess at the categories of interest, and please re-organize change as you see fit. I'm not totally happy with the navigation (the sidebar crunches both h2 and h3 together) but we can fix that up when the page content is all there and we know how much there is :)

I don't have any other major updates - I'm mid-development of the details pages (starting skeleton work here) and I'll post updates on this issue. It's quite a bit of frontend work (and backend too) so likely I'll need a few days and work on it into the weekend. I'll keep you updated! Going to post some other additions to the documentation page in the linked issue now.

vsoch commented 5 years ago

actually don't update the FAQ just yet - I'm going to fix the nesting issue (it's bothering me already lol)

vsoch commented 5 years ago

okay all done -> https://vsoch.github.io/freegenes/docs/getting-started/faq edit away! I'm closing up shop soon so I won't get in your hair :haircut_man: :haircut_woman:

hverdonk commented 5 years ago

The FAQ page looks great, thank you!

For the "backbone" question, after a talk with Keoni about how he structured the original database, I have a better way of explaining what I mean. If for a given part, the "part_type" field is "vector", then we would want to return the information stored in that part's "optimized_sequence" (or possibly it's full sequence. You'll have to ask Keoni which he prefers)

vsoch commented 5 years ago

Noted! This also sounds like one of the details that we will get to once we have the detail pages finished, which I’m currently working on. Will keep you updated.

vsoch commented 5 years ago

@Koeng101 and @hverdonk the shipping API is implemented, including the whole flow for signing and receiving the MTA. We store the result from shippo for the label and transaction, but with the billing field removed, and we don't store any addresses. If this issue is resolved, please feel free to close! And @hverdonk please go ahead and open a PR to add content to the FAQ page. Would you like me to open a separate issue for this?